[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:57 dist-test-slave-cb7x krb5kdc[7252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:57 dist-test-slave-cb7x krb5kdc[7252](info): set up 2 sockets
Sep 28 12:25:57 dist-test-slave-cb7x krb5kdc[7252](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 12:25:58.737484 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:25:58 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:25:58.855846 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34163
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:34163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:58.934033 7265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:58.934264 7265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:58.934299 7265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:58.935472 7265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:25:58.935544 7265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:58.935567 7265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:58.935590 7265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:25:58.935619 7265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:25:58.937122 7265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:34163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:58.937407 7265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:58.937647 7265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:58.942508 7273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:58.942509 7270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:58.942626 7271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:58.942934 7265 server_base.cc:1034] running on GCE node
I20260928 12:25:58.943216 7265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:58.943552 7265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:58.944738 7265 hybrid_clock.cc:648] HybridClock initialized: now 1790598358944720 us; error 35 us; skew 500 ppm
Sep 28 12:25:58 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598358, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:58.947165 7265 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:25:58.947721 7265 webserver.cc:460] Webserver started at http://127.7.19.126:37645/ using document root <none> and password file <none>
I20260928 12:25:58.947966 7265 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:58.948035 7265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:58.948187 7265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:58.949309 7265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "5e88c74e5ee64b9fb1451201365f39af"
format_stamp: "Formatted at 2026-09-28 12:25:58 on dist-test-slave-cb7x"
server_key: "5f512869e7249b8d76a54a9d614e94df"
server_key_iv: "57ba5783ee4846388dce7fe59c7c4aa0"
server_key_version: "encryptionkey@0"
I20260928 12:25:58.949671 7265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "5e88c74e5ee64b9fb1451201365f39af"
format_stamp: "Formatted at 2026-09-28 12:25:58 on dist-test-slave-cb7x"
server_key: "5f512869e7249b8d76a54a9d614e94df"
server_key_iv: "57ba5783ee4846388dce7fe59c7c4aa0"
server_key_version: "encryptionkey@0"
I20260928 12:25:58.951020 7265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:58.951726 7280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:58.951927 7265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:58.952032 7265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "5e88c74e5ee64b9fb1451201365f39af"
format_stamp: "Formatted at 2026-09-28 12:25:58 on dist-test-slave-cb7x"
server_key: "5f512869e7249b8d76a54a9d614e94df"
server_key_iv: "57ba5783ee4846388dce7fe59c7c4aa0"
server_key_version: "encryptionkey@0"
I20260928 12:25:58.952105 7265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:58.965461 7265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:58.967371 7265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:58.967533 7265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:58.971496 7265 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:34163
I20260928 12:25:58.971531 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:34163 every 8 connection(s)
I20260928 12:25:58.971865 7265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:25:58.972795 7333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:25:58.974915 7333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af: Bootstrap starting.
I20260928 12:25:58.975441 7333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af: Neither blocks nor log segments found. Creating new log.
I20260928 12:25:58.975682 7333 log.cc:824] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af: Log is configured to *not* fsync() on all Append() calls
I20260928 12:25:58.976313 7333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af: No bootstrap required, opened a new log
I20260928 12:25:58.977917 7333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e88c74e5ee64b9fb1451201365f39af" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34163 } }
I20260928 12:25:58.978066 7333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:25:58.978091 7333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e88c74e5ee64b9fb1451201365f39af, State: Initialized, Role: FOLLOWER
I20260928 12:25:58.978246 7333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e88c74e5ee64b9fb1451201365f39af" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34163 } }
I20260928 12:25:58.978303 7333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:25:58.978323 7333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:25:58.978355 7333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:25:58.978799 7333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e88c74e5ee64b9fb1451201365f39af" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34163 } }
I20260928 12:25:58.978883 7333 leader_election.cc:302] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e88c74e5ee64b9fb1451201365f39af; no voters:
I20260928 12:25:58.978995 7333 leader_election.cc:288] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:25:58.979058 7336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:25:58.979266 7333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:25:58.979252 7336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [term 1 LEADER]: Becoming Leader. State: Replica: 5e88c74e5ee64b9fb1451201365f39af, State: Running, Role: LEADER
I20260928 12:25:58.979507 7336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e88c74e5ee64b9fb1451201365f39af" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34163 } }
I20260928 12:25:58.980159 7338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e88c74e5ee64b9fb1451201365f39af. Latest consensus state: current_term: 1 leader_uuid: "5e88c74e5ee64b9fb1451201365f39af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e88c74e5ee64b9fb1451201365f39af" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34163 } } }
I20260928 12:25:58.980171 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e88c74e5ee64b9fb1451201365f39af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e88c74e5ee64b9fb1451201365f39af" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34163 } } }
I20260928 12:25:58.980260 7338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:58.980278 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af [sys.catalog]: This master's current role is: LEADER
I20260928 12:25:58.980492 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 7265
I20260928 12:25:58.980556 7345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:25:58.980603 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:25:58.981115 7345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:25:58.981503 7245 external_mini_cluster.cc:1428] Setting key 757b0243cd0eb1a75c8f60b74b64bef5
I20260928 12:25:58.982935 7345 catalog_manager.cc:1347] Generated new cluster ID: 7b866d5ec44747f78e87d1fcd573c7ec
I20260928 12:25:58.982990 7345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:25:58 dist-test-slave-cb7x krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:25:59.002859 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:58.982620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.7.19.126:34163)
0928 12:25:58.982859 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:58.982868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:58.982901 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:25:58.990611 (+ 7710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:58.990616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:58.990667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:58.990801 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:58.992356 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:58.993557 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:58.994629 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:58.995027 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:58.995145 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:58.998705 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:58.998710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:58.998710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:58.998728 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:59.000396 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.001158 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.001161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.001161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.001303 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.001704 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.001706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.001706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.001980 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:59.002203 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:59.002452 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 12:25:59.002580 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260928 12:25:59.023612 7345 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:25:59.024163 7345 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:25:59.035281 7345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e88c74e5ee64b9fb1451201365f39af: Generated new TSK 0
I20260928 12:25:59.035501 7345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:25:59.063410 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34163
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:59.147475 7359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:59.147670 7359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:59.147694 7359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:59.148869 7359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:59.148914 7359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:59.148948 7359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:25:59.150380 7359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:59.150610 7359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:59.150847 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:59.155772 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:59.155838 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:59.155846 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:59.156049 7359 server_base.cc:1034] running on GCE node
I20260928 12:25:59.156350 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:59.156538 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:59.157678 7359 hybrid_clock.cc:648] HybridClock initialized: now 1790598359157657 us; error 36 us; skew 500 ppm
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:59.160197 7359 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:25:59.160614 7359 webserver.cc:460] Webserver started at http://127.7.19.65:33379/ using document root <none> and password file <none>
I20260928 12:25:59.160876 7359 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:59.160923 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:59.161072 7359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:59.162120 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "d07c3ae5e8224767ab858b34b9603dde"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "d543749030e34c5e2e691c45df77126b"
server_key_iv: "c1c917db4b718ea76212ddd0dd0576fe"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.162490 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "d07c3ae5e8224767ab858b34b9603dde"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "d543749030e34c5e2e691c45df77126b"
server_key_iv: "c1c917db4b718ea76212ddd0dd0576fe"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.163762 7359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:25:59.164561 7374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.164803 7359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:59.164908 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "d07c3ae5e8224767ab858b34b9603dde"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "d543749030e34c5e2e691c45df77126b"
server_key_iv: "c1c917db4b718ea76212ddd0dd0576fe"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.164990 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:59.197500 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:59.200242 7359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:59.200426 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:59.200727 7359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:59.201081 7359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:59.201118 7359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.201165 7359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:59.201193 7359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.207147 7359 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:41183
I20260928 12:25:59.207221 7487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:41183 every 8 connection(s)
I20260928 12:25:59.207579 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:25:59.208847 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 7359
I20260928 12:25:59.208927 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:25:59.209045 7245 external_mini_cluster.cc:1428] Setting key ff695eba1ac966740443366ff55d3841
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
I20260928 12:25:59.218763 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.208307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:39793 (local address 127.7.19.126:34163)
0928 12:25:59.208373 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:59.208375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:59.208597 (+ 222us) server_negotiation.cc:407] Connection header received
0928 12:25:59.209323 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:59.209324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:59.209340 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:59.209413 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:59.210527 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.211297 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.212556 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.212985 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.213078 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.215202 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:59.215203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:59.215204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:59.215212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:59.217006 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.217499 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.217501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.217501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.217570 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.218000 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.218001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.218001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.218146 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:59.218211 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:59.218603 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 12:25:59.218694 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:25:59.218902 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.208298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34163 (local address 127.7.19.65:39793)
0928 12:25:59.208479 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:59.208513 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:59.209188 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:59.209514 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:59.209519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:59.210220 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:59.210420 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:59.210425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.211416 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.211417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:59.212461 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:59.212464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.213088 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.213090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:59.213219 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.213748 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:59.213757 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:59.215096 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:59.217113 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:59.217119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:59.217121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:59.217412 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:59.217787 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:59.217788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:59.217789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:59.217923 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:59.218259 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:59.218264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:59.218432 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 12:25:59.218542 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:25:59.218665 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260928 12:25:59.219544 7488 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34163
I20260928 12:25:59.219671 7488 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:59.219857 7488 heartbeater.cc:511] Master 127.7.19.126:34163 requested a full tablet report, sending...
I20260928 12:25:59.220465 7297 ts_manager.cc:194] Registered new tserver with Master: d07c3ae5e8224767ab858b34b9603dde (127.7.19.65:41183)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
I20260928 12:25:59.221247 7297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:39793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:25:59.239194 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34163
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:59.319589 7493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:59.319820 7493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:59.319856 7493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:59.321013 7493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:59.321080 7493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:59.321128 7493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:25:59.322543 7493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:59.322803 7493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:59.323025 7493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:59.328076 7498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:59.328303 7493 server_base.cc:1034] running on GCE node
W20260928 12:25:59.328076 7501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:59.328114 7499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:59.328657 7493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:59.328960 7493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:59.330122 7493 hybrid_clock.cc:648] HybridClock initialized: now 1790598359330113 us; error 26 us; skew 500 ppm
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:59.333134 7493 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:25:59.333643 7493 webserver.cc:460] Webserver started at http://127.7.19.66:34083/ using document root <none> and password file <none>
I20260928 12:25:59.333870 7493 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:59.333936 7493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:59.334070 7493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:59.335156 7493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "29265004ca7249678e54294e3821c38a"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "1b998abbb2ba8af160bd04e866da397b"
server_key_iv: "ed0bc646f5f024f4824044c7ffdb7722"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.335533 7493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "29265004ca7249678e54294e3821c38a"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "1b998abbb2ba8af160bd04e866da397b"
server_key_iv: "ed0bc646f5f024f4824044c7ffdb7722"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.336867 7493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:25:59.337594 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.337765 7493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:25:59.337873 7493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "29265004ca7249678e54294e3821c38a"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "1b998abbb2ba8af160bd04e866da397b"
server_key_iv: "ed0bc646f5f024f4824044c7ffdb7722"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.337944 7493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:59.353367 7493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:59.355641 7493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:59.355789 7493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:59.356003 7493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:59.356302 7493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:59.356362 7493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.356405 7493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:59.356433 7493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.362190 7493 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:38311
I20260928 12:25:59.362231 7621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:38311 every 8 connection(s)
I20260928 12:25:59.362555 7493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:25:59.364094 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 7493
I20260928 12:25:59.364188 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:25:59.364341 7245 external_mini_cluster.cc:1428] Setting key 31b3a0919890a0db4a972ec24cf01351
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:25:59.373425 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.363170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:59175 (local address 127.7.19.126:34163)
0928 12:25:59.363230 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:59.363232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:59.363429 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:25:59.364578 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:59.364580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:59.364592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:59.364656 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:59.365849 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.366336 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.367344 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.367568 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.367648 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.370304 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:59.370305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:59.370306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:59.370313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:59.371939 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.372400 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.372401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.372401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.372454 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.372780 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.372781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.372781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.372943 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:59.372991 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:59.373285 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 12:25:59.373358 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:25:59.373538 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.363173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34163 (local address 127.7.19.66:59175)
0928 12:25:59.363312 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:59.363344 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:59.364421 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:59.364749 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:59.364752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:59.365484 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:59.365755 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:59.365761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.366444 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.366445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:59.367249 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:59.367251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.367655 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.367656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:59.367782 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.368327 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:59.368335 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:59.370198 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:59.372024 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:59.372031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:59.372033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:59.372322 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:59.372521 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:59.372522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:59.372522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:59.372662 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:59.373047 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:59.373051 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:59.373160 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 12:25:59.373239 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:25:59.373356 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260928 12:25:59.374087 7622 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34163
I20260928 12:25:59.374171 7622 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:59.374348 7622 heartbeater.cc:511] Master 127.7.19.126:34163 requested a full tablet report, sending...
I20260928 12:25:59.374743 7297 ts_manager.cc:194] Registered new tserver with Master: 29265004ca7249678e54294e3821c38a (127.7.19.66:38311)
I20260928 12:25:59.375195 7297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:59175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:25:59.390995 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34163
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:25:59.472107 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:25:59.472311 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:25:59.472334 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:25:59.473484 7627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:25:59.473526 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:25:59.473552 7627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:25:59.474951 7627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:25:59.475183 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:25:59.475405 7627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:25:59.480338 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:59.480453 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:25:59.480340 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:25:59.480546 7627 server_base.cc:1034] running on GCE node
I20260928 12:25:59.480823 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:25:59.481069 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:25:59.482255 7627 hybrid_clock.cc:648] HybridClock initialized: now 1790598359482200 us; error 48 us; skew 500 ppm
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:25:59.484799 7627 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:25:59.485240 7627 webserver.cc:460] Webserver started at http://127.7.19.67:37371/ using document root <none> and password file <none>
I20260928 12:25:59.485419 7627 fs_manager.cc:359] Metadata directory not provided
I20260928 12:25:59.485456 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:25:59.485543 7627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:25:59.486511 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "86feedef77a8495091bc98adb5f8a17e"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "3d131aa7271a141d451cc54a2087fee2"
server_key_iv: "e5dba8b9a4ecbc269bcd26e70406d89b"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.486809 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "86feedef77a8495091bc98adb5f8a17e"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "3d131aa7271a141d451cc54a2087fee2"
server_key_iv: "e5dba8b9a4ecbc269bcd26e70406d89b"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.488080 7627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:25:59.488863 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.489050 7627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:25:59.489147 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "86feedef77a8495091bc98adb5f8a17e"
format_stamp: "Formatted at 2026-09-28 12:25:59 on dist-test-slave-cb7x"
server_key: "3d131aa7271a141d451cc54a2087fee2"
server_key_iv: "e5dba8b9a4ecbc269bcd26e70406d89b"
server_key_version: "encryptionkey@0"
I20260928 12:25:59.489235 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:25:59.533157 7627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:25:59.535632 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:25:59.535815 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:25:59.536051 7627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:25:59.536367 7627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:25:59.536418 7627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.536461 7627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:25:59.536489 7627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:25:59.542748 7627 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:33583
I20260928 12:25:59.542821 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:33583 every 8 connection(s)
I20260928 12:25:59.543133 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:25:59.545957 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 7627
I20260928 12:25:59.546017 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:25:59.546168 7245 external_mini_cluster.cc:1428] Setting key 1739308d0d303e376f36ef600aadd4c8
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:25:59.552235 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.543818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:57325 (local address 127.7.19.126:34163)
0928 12:25:59.543859 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:59.543861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:59.544048 (+ 187us) server_negotiation.cc:407] Connection header received
0928 12:25:59.544524 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:59.544525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:59.544534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:59.544594 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:59.545526 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.546033 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.547396 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.547591 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.547671 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.549782 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:59.549783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:59.549784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:59.549791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:59.550943 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.551320 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.551321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.551322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.551380 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.551617 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.551618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.551618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.551729 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:59.551802 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:59.552068 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 12:25:59.552144 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:25:59.552415 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.543788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34163 (local address 127.7.19.67:57325)
0928 12:25:59.543937 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:25:59.543978 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:25:59.544444 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:25:59.544649 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:25:59.544652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:25:59.545245 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:25:59.545445 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:59.545449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.546135 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.546137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:59.547309 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:25:59.547312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.547679 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.547680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:25:59.547808 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.548232 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:25:59.548240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:25:59.549685 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:25:59.551022 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:59.551028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:59.551030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:59.551245 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:59.551444 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:25:59.551445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:25:59.551446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:25:59.551552 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:25:59.551792 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:25:59.551798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:25:59.551933 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 12:25:59.552033 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:25:59.552181 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260928 12:25:59.552867 7756 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34163
I20260928 12:25:59.552950 7756 heartbeater.cc:464] Registering TS with master...
I20260928 12:25:59.553082 7756 heartbeater.cc:511] Master 127.7.19.126:34163 requested a full tablet report, sending...
I20260928 12:25:59.553417 7297 ts_manager.cc:194] Registered new tserver with Master: 86feedef77a8495091bc98adb5f8a17e (127.7.19.67:33583)
I20260928 12:25:59.553768 7297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:57325
I20260928 12:25:59.558050 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:25:59.582355 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.573140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.7.19.65:41183)
0928 12:25:59.573304 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:59.573308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:59.573318 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:25:59.573343 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:59.573345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:59.573359 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:59.573430 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:59.573934 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.574647 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.575333 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.575628 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.575679 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.578468 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:59.578477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:59.578477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:59.578485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:59.580559 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.581048 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.581052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.581053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.581115 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.581413 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.581414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.581414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.581602 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:59.581735 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:59.582171 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 12:25:59.582249 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260928 12:25:59.582933 7402 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33972
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:25:59.606902 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:25:59.599976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.7.19.65:41183)
0928 12:25:59.600036 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:25:59.600039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:25:59.600101 (+ 62us) server_negotiation.cc:407] Connection header received
0928 12:25:59.600207 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:25:59.600208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:25:59.600221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:25:59.600288 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:25:59.600891 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.601375 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.602125 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:25:59.602328 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:25:59.602408 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:25:59.604465 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:25:59.604466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:25:59.604466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:25:59.604474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:25:59.605763 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.606151 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.606152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.606153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.606214 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:25:59.606411 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:25:59.606412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:25:59.606413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:25:59.606510 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:25:59.606632 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 12:25:59.606719 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:25:59.606798 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:25:59.607573 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 7359
I20260928 12:25:59.612591 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 7493
I20260928 12:25:59.618034 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 7627
I20260928 12:25:59.622156 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 7265
2026-09-28T12:25:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1930 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7770](info): set up 2 sockets
Sep 28 12:25:59 dist-test-slave-cb7x krb5kdc[7770](info): commencing operation
krb5kdc: starting...
W20260928 12:26:00.656236 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:00 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:00.771481 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46071
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:00.850553 7783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:00.850754 7783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:00.850775 7783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:00.851846 7783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:00.851882 7783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:00.851897 7783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:00.851909 7783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:00.851922 7783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:00.853291 7783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:00.853482 7783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:00.853646 7783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:00.858233 7788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:00.858219 7791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:00.858219 7789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:00.858368 7783 server_base.cc:1034] running on GCE node
I20260928 12:26:00.858759 7783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:00.859030 7783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:00.860217 7783 hybrid_clock.cc:648] HybridClock initialized: now 1790598360860209 us; error 33 us; skew 500 ppm
Sep 28 12:26:00 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598360, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:00.862711 7783 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:00.863121 7783 webserver.cc:460] Webserver started at http://127.7.19.126:39545/ using document root <none> and password file <none>
I20260928 12:26:00.863297 7783 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:00.863343 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:00.863428 7783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:00.864419 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "debc73ac124a4e5c8633b4e3fe321831"
format_stamp: "Formatted at 2026-09-28 12:26:00 on dist-test-slave-cb7x"
server_key: "ae7a7e7288930812e83bb44202fb6067"
server_key_iv: "fb334421dd00d64336e4ded355f58ad1"
server_key_version: "encryptionkey@0"
I20260928 12:26:00.864785 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "debc73ac124a4e5c8633b4e3fe321831"
format_stamp: "Formatted at 2026-09-28 12:26:00 on dist-test-slave-cb7x"
server_key: "ae7a7e7288930812e83bb44202fb6067"
server_key_iv: "fb334421dd00d64336e4ded355f58ad1"
server_key_version: "encryptionkey@0"
I20260928 12:26:00.866010 7783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:00.866679 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:00.866846 7783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:00.866930 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "debc73ac124a4e5c8633b4e3fe321831"
format_stamp: "Formatted at 2026-09-28 12:26:00 on dist-test-slave-cb7x"
server_key: "ae7a7e7288930812e83bb44202fb6067"
server_key_iv: "fb334421dd00d64336e4ded355f58ad1"
server_key_version: "encryptionkey@0"
I20260928 12:26:00.866995 7783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:00.876196 7783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:00.878290 7783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:00.878448 7783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:00.882246 7783 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:46071
I20260928 12:26:00.882311 7850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:46071 every 8 connection(s)
I20260928 12:26:00.882618 7783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:00.883443 7851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:00.885571 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 7783
I20260928 12:26:00.885650 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:00.885654 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831: Bootstrap starting.
I20260928 12:26:00.885821 7245 external_mini_cluster.cc:1428] Setting key 84505458a2b92238c2119e6828d14a4d
I20260928 12:26:00.886344 7851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:00.886631 7851 log.cc:824] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:00.887274 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831: No bootstrap required, opened a new log
I20260928 12:26:00.889088 7851 raft_consensus.cc:348] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc73ac124a4e5c8633b4e3fe321831" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46071 } }
I20260928 12:26:00.889230 7851 raft_consensus.cc:374] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:00.889284 7851 raft_consensus.cc:729] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debc73ac124a4e5c8633b4e3fe321831, State: Initialized, Role: FOLLOWER
I20260928 12:26:00.889405 7851 consensus_queue.cc:260] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc73ac124a4e5c8633b4e3fe321831" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46071 } }
I20260928 12:26:00.889490 7851 raft_consensus.cc:388] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:00.889533 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:00.889571 7851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:00.890062 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc73ac124a4e5c8633b4e3fe321831" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46071 } }
I20260928 12:26:00.890173 7851 leader_election.cc:302] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debc73ac124a4e5c8633b4e3fe321831; no voters:
I20260928 12:26:00.890352 7851 leader_election.cc:288] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:00.890407 7856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:00.890611 7851 sys_catalog.cc:564] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:00.891525 7856 raft_consensus.cc:686] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [term 1 LEADER]: Becoming Leader. State: Replica: debc73ac124a4e5c8633b4e3fe321831, State: Running, Role: LEADER
I20260928 12:26:00.891616 7856 consensus_queue.cc:237] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc73ac124a4e5c8633b4e3fe321831" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46071 } }
I20260928 12:26:00.891945 7864 sys_catalog.cc:455] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader debc73ac124a4e5c8633b4e3fe321831. Latest consensus state: current_term: 1 leader_uuid: "debc73ac124a4e5c8633b4e3fe321831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc73ac124a4e5c8633b4e3fe321831" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46071 } } }
Sep 28 12:26:00 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:00.892122 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "debc73ac124a4e5c8633b4e3fe321831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc73ac124a4e5c8633b4e3fe321831" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46071 } } }
I20260928 12:26:00.892468 7864 sys_catalog.cc:458] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:00.892693 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831 [sys.catalog]: This master's current role is: LEADER
W20260928 12:26:00.893087 7870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:26:00.893146 7870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:00.893208 7872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:00.893352 7872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:00.895222 7872 catalog_manager.cc:1347] Generated new cluster ID: d352f71269ab49eb94c1487b024587fc
I20260928 12:26:00.895267 7872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:00.896133 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:00.886314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.7.19.126:46071)
0928 12:26:00.886510 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:00.886518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:00.886549 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:26:00.886938 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:00.886945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:00.886987 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:00.887134 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:00.887770 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:00.888630 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:00.889463 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:00.889947 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:00.890058 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:00.892541 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:00.892545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:00.892545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:00.892558 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:00.894577 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:00.894916 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:00.894919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:00.894920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:00.895021 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:00.895237 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:00.895237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:00.895238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:00.895513 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:00.895663 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:00.895794 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 12:26:00.895932 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260928 12:26:00.909350 7872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:00.909901 7872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:00.919865 7872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debc73ac124a4e5c8633b4e3fe321831: Generated new TSK 0
I20260928 12:26:00.920049 7872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:00.951646 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46071
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:01.048768 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:01.048979 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:01.049005 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:01.050239 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:01.050279 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:01.050307 7877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:01.051723 7877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:01.051904 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:01.052079 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:01.056838 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:01.056860 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:01.056962 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:01.057201 7877 server_base.cc:1034] running on GCE node
I20260928 12:26:01.057463 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:01.057701 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:01.058857 7877 hybrid_clock.cc:648] HybridClock initialized: now 1790598361058848 us; error 29 us; skew 500 ppm
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:01.061368 7877 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:01.061842 7877 webserver.cc:460] Webserver started at http://127.7.19.65:33269/ using document root <none> and password file <none>
I20260928 12:26:01.062067 7877 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:01.062131 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:01.062263 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:01.063284 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "135d121d4e4641468342b70f2cc254c0"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "e7dbe163fb347ff763c32c63669cd9f8"
server_key_iv: "bebb435495c85b55b3cf29a6fc9403a0"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.063685 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "135d121d4e4641468342b70f2cc254c0"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "e7dbe163fb347ff763c32c63669cd9f8"
server_key_iv: "bebb435495c85b55b3cf29a6fc9403a0"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.065009 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:01.065735 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.065917 7877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:01.066015 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "135d121d4e4641468342b70f2cc254c0"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "e7dbe163fb347ff763c32c63669cd9f8"
server_key_iv: "bebb435495c85b55b3cf29a6fc9403a0"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.066083 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:01.091711 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:01.093827 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:01.093992 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:01.094228 7877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:01.094534 7877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:01.094583 7877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.094619 7877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:01.094645 7877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.100201 7877 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:39809
I20260928 12:26:01.100270 8005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:39809 every 8 connection(s)
I20260928 12:26:01.100603 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:01.106948 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 7877
I20260928 12:26:01.107059 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:01.107235 7245 external_mini_cluster.cc:1428] Setting key cdf1cb49d11e55dd49e906494cb6f3d2
I20260928 12:26:01.110220 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.101340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:58489 (local address 127.7.19.126:46071)
0928 12:26:01.101393 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.101395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.101567 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:26:01.102511 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.102512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.102523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.102590 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.103180 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.103688 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.104651 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.104890 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.104952 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.107004 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.107006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.107006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.107013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.108877 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.109282 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.109283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.109284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.109345 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.109586 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.109587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.109588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.109702 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.109771 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.110038 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.110128 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:01.110417 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.101306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46071 (local address 127.7.19.65:58489)
0928 12:26:01.101457 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.101499 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.102398 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.102654 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.102657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.102905 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:01.103090 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.103094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.103767 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.103768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.104566 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.104567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.104964 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.104965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.105064 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.105572 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:01.105580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:01.106914 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:01.108950 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:01.108955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:01.108957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:01.109195 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:01.109412 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:01.109413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:01.109413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:01.109521 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:01.109776 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:01.109783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:01.109916 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.110021 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.110173 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260928 12:26:01.110884 8006 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46071
I20260928 12:26:01.110987 8006 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:01.111152 8006 heartbeater.cc:511] Master 127.7.19.126:46071 requested a full tablet report, sending...
I20260928 12:26:01.111634 7815 ts_manager.cc:194] Registered new tserver with Master: 135d121d4e4641468342b70f2cc254c0 (127.7.19.65:39809)
I20260928 12:26:01.112398 7815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:58489
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:01.131990 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46071
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:01.210325 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:01.210511 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:01.210534 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:01.211696 8011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:01.211758 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:01.211807 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:01.213263 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:01.213438 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:01.213599 8011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:01.218617 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:01.218636 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:01.218649 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:01.218667 8011 server_base.cc:1034] running on GCE node
I20260928 12:26:01.219066 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:01.219311 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:01.220459 8011 hybrid_clock.cc:648] HybridClock initialized: now 1790598361220448 us; error 31 us; skew 500 ppm
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:01.222936 8011 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:01.223385 8011 webserver.cc:460] Webserver started at http://127.7.19.66:43301/ using document root <none> and password file <none>
I20260928 12:26:01.223606 8011 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:01.223670 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:01.223814 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:01.224923 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "9d35fc052a2e4da3bb822057b8a455fd"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "00e05e24232f1f0942d1892003b7611c"
server_key_iv: "2c9dd031bf8647a972df8e65c5f1697e"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.225270 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "9d35fc052a2e4da3bb822057b8a455fd"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "00e05e24232f1f0942d1892003b7611c"
server_key_iv: "2c9dd031bf8647a972df8e65c5f1697e"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.226660 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:26:01.227329 8026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.227488 8011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.227583 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "9d35fc052a2e4da3bb822057b8a455fd"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "00e05e24232f1f0942d1892003b7611c"
server_key_iv: "2c9dd031bf8647a972df8e65c5f1697e"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.227650 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:01.251451 8011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:01.253786 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:01.253939 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:01.254179 8011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:01.254483 8011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:01.254534 8011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.254575 8011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:01.254608 8011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.260818 8011 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:39245
I20260928 12:26:01.260893 8139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:39245 every 8 connection(s)
I20260928 12:26:01.261219 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:01.266511 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8011
I20260928 12:26:01.266642 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:01.266815 7245 external_mini_cluster.cc:1428] Setting key 2aca740e0905352368fba30a299d4b36
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:01.271306 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.262036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:40773 (local address 127.7.19.126:46071)
0928 12:26:01.262121 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.262123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.262349 (+ 226us) server_negotiation.cc:407] Connection header received
0928 12:26:01.263217 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.263218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.263228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.263292 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.263946 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.264439 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.265524 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.265709 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.265768 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.268171 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.268172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.268173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.268179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.269793 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.270233 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.270235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.270235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.270298 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.270614 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.270615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.270615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.270757 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.270807 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.271165 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.271241 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:26:01.271600 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.261963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46071 (local address 127.7.19.66:40773)
0928 12:26:01.262211 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.262242 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.263129 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.263349 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.263352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.263604 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:01.263805 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.263810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.264531 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.264531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.265431 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.265433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.265783 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.265784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.265885 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.266390 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:01.266398 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:01.268076 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:01.269888 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:01.269897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:01.269900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:01.270144 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:01.270380 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:01.270381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:01.270381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:01.270540 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:01.270822 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:01.270828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:01.271010 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.271140 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.271337 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260928 12:26:01.272116 8140 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46071
I20260928 12:26:01.272210 8140 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:01.272408 8140 heartbeater.cc:511] Master 127.7.19.126:46071 requested a full tablet report, sending...
I20260928 12:26:01.272807 7815 ts_manager.cc:194] Registered new tserver with Master: 9d35fc052a2e4da3bb822057b8a455fd (127.7.19.66:39245)
I20260928 12:26:01.273183 7815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:40773
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:01.291996 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46071
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:01.371840 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:01.372082 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:01.372120 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:01.373301 8145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:01.373365 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:01.373409 8145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:01.374877 8145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46071
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:01.375151 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:01.375370 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:01.380256 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:01.380270 8153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:01.380288 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:01.380503 8145 server_base.cc:1034] running on GCE node
I20260928 12:26:01.380772 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:01.381017 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:01.382186 8145 hybrid_clock.cc:648] HybridClock initialized: now 1790598361382176 us; error 38 us; skew 500 ppm
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:01.384647 8145 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:01.385089 8145 webserver.cc:460] Webserver started at http://127.7.19.67:42877/ using document root <none> and password file <none>
I20260928 12:26:01.385324 8145 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:01.385389 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:01.385524 8145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:01.386586 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "d338d96f80d24a3cbee0bb8133c83b9d"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "8392d106e42a2e5691503db742d1c370"
server_key_iv: "fd44c159243e6ff7a03be5c2b96471ac"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.386946 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "d338d96f80d24a3cbee0bb8133c83b9d"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "8392d106e42a2e5691503db742d1c370"
server_key_iv: "fd44c159243e6ff7a03be5c2b96471ac"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.388208 8145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:01.388957 8160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.389111 8145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:01.389204 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "d338d96f80d24a3cbee0bb8133c83b9d"
format_stamp: "Formatted at 2026-09-28 12:26:01 on dist-test-slave-cb7x"
server_key: "8392d106e42a2e5691503db742d1c370"
server_key_iv: "fd44c159243e6ff7a03be5c2b96471ac"
server_key_version: "encryptionkey@0"
I20260928 12:26:01.389276 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:01.399109 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:01.401264 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:01.401412 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:01.401620 8145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:01.401897 8145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:01.401947 8145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.401986 8145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:01.402011 8145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:01.407447 8145 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:36963
I20260928 12:26:01.407521 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:36963 every 8 connection(s)
I20260928 12:26:01.407819 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:01.416343 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8145
I20260928 12:26:01.416469 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:01.416530 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.408492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:58991 (local address 127.7.19.126:46071)
0928 12:26:01.408555 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.408558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.408782 (+ 224us) server_negotiation.cc:407] Connection header received
0928 12:26:01.409695 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.409695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.409705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.409763 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.410315 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.410812 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.411780 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.411972 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.412024 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.414035 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.414037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.414037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.414043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.415344 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.415693 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.415693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.415694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.415741 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.415944 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.415945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.415945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.416058 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.416102 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.416333 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.416411 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:26:01.416642 7245 external_mini_cluster.cc:1428] Setting key a9b8fb2cce00047cbb7a179d68fbe95a
I20260928 12:26:01.416751 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.408482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46071 (local address 127.7.19.67:58991)
0928 12:26:01.408630 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.408694 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.409604 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.409816 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.409819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.410052 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:01.410246 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.410250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.410885 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.410886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.411707 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.411709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.412036 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.412037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.412131 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.412638 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:01.412646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:01.413951 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:01.415414 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:01.415419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:01.415421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:01.415628 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:01.415795 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:01.415796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:01.415796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:01.415891 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:01.416103 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:01.416106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:01.416214 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.416290 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.416469 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 12:26:01.417301 8274 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46071
I20260928 12:26:01.417398 8274 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:01.417554 8274 heartbeater.cc:511] Master 127.7.19.126:46071 requested a full tablet report, sending...
I20260928 12:26:01.417893 7815 ts_manager.cc:194] Registered new tserver with Master: d338d96f80d24a3cbee0bb8133c83b9d (127.7.19.67:36963)
I20260928 12:26:01.418255 7815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:58991
I20260928 12:26:01.428452 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:26:01.435959 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.429934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.7.19.126:46071)
0928 12:26:01.429996 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.429998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.430046 (+ 48us) server_negotiation.cc:407] Connection header received
0928 12:26:01.430096 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.430097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.430105 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.430150 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.430731 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.431204 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.432335 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.432650 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.432745 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.433867 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.433867 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.433868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.433872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.434992 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.435318 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.435319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.435320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.435366 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.435513 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.435514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.435514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.435641 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.435713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.435800 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.435856 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:26:01.438033 7815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:01.439574 7815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:01.445941 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.442032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.7.19.66:39245)
0928 12:26:01.442229 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.442235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.442257 (+ 22us) server_negotiation.cc:407] Connection header received
0928 12:26:01.442300 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.442303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.442318 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.442610 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:01.442683 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.443776 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445027 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.445510 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445604 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.445662 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.445820 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.445862 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260928 12:26:01.445935 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.441907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39245 (local address 127.0.0.1:45110)
0928 12:26:01.442073 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.442110 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.442171 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.442461 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.442463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.442466 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:01.442531 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.442533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.443887 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.443888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.444949 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.444951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.445606 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.445694 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.445707 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.445801 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.445860 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260928 12:26:01.446072 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.442229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.7.19.65:39809)
0928 12:26:01.442402 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.442407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.442438 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:26:01.442508 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.442510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.442524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.442600 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:01.442811 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.443828 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.444880 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.445594 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445662 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.445712 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.445964 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.445999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260928 12:26:01.446086 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.442237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.7.19.67:36963)
0928 12:26:01.442419 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.442423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.442433 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:01.442472 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.442474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.442491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.442598 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:01.442775 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.443473 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.444892 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.445593 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445662 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.445712 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.445979 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.446015 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260928 12:26:01.446234 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.441907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36963 (local address 127.0.0.1:42364)
0928 12:26:01.442264 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.442274 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.442310 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.442563 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.442564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.442566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:01.442618 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.442620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.443572 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.443573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.444818 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.444820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.445749 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.445819 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.445825 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.445885 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.445922 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":9,"thread_start_us":61,"threads_started":1}
I20260928 12:26:01.446319 8288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.442111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39809 (local address 127.0.0.1:56842)
0928 12:26:01.442390 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.442399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.442446 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.442681 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.442681 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.442683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:01.442724 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.442725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.443924 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.443925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.444795 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.444797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.445755 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.445756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.445819 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.445825 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.445876 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.445922 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 12:26:01.447039 7940 tablet_service.cc:1467] Processing CreateTablet for tablet a9ea3d9d6c1649c0bdb9f242acac531a (DEFAULT_TABLE table=test-table [id=221c3e4d699445909244a22575db3cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:01.447106 8208 tablet_service.cc:1467] Processing CreateTablet for tablet a9ea3d9d6c1649c0bdb9f242acac531a (DEFAULT_TABLE table=test-table [id=221c3e4d699445909244a22575db3cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:01.447410 7940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ea3d9d6c1649c0bdb9f242acac531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:01.447425 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ea3d9d6c1649c0bdb9f242acac531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:01.449301 8074 tablet_service.cc:1467] Processing CreateTablet for tablet a9ea3d9d6c1649c0bdb9f242acac531a (DEFAULT_TABLE table=test-table [id=221c3e4d699445909244a22575db3cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:01.449596 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ea3d9d6c1649c0bdb9f242acac531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:01.450235 8292 tablet_bootstrap.cc:492] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: Bootstrap starting.
I20260928 12:26:01.450881 8292 tablet_bootstrap.cc:654] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:01.451467 8292 log.cc:824] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:01.452199 8292 tablet_bootstrap.cc:492] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: No bootstrap required, opened a new log
I20260928 12:26:01.452275 8292 ts_tablet_manager.cc:1397] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:01.452822 8291 tablet_bootstrap.cc:492] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: Bootstrap starting.
I20260928 12:26:01.452822 8294 tablet_bootstrap.cc:492] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: Bootstrap starting.
I20260928 12:26:01.453512 8294 tablet_bootstrap.cc:654] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:01.453505 8291 tablet_bootstrap.cc:654] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:01.454063 8292 raft_consensus.cc:348] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.454162 8291 log.cc:824] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:01.454200 8294 log.cc:824] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:01.454216 8292 raft_consensus.cc:374] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:01.454252 8292 raft_consensus.cc:729] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135d121d4e4641468342b70f2cc254c0, State: Initialized, Role: FOLLOWER
I20260928 12:26:01.454358 8292 consensus_queue.cc:260] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.454655 8292 ts_tablet_manager.cc:1428] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:26:01.454784 8006 heartbeater.cc:503] Master 127.7.19.126:46071 was elected leader, sending a full tablet report...
I20260928 12:26:01.455137 8291 tablet_bootstrap.cc:492] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: No bootstrap required, opened a new log
I20260928 12:26:01.455204 8291 ts_tablet_manager.cc:1397] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:01.455080 8294 tablet_bootstrap.cc:492] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: No bootstrap required, opened a new log
I20260928 12:26:01.455443 8294 ts_tablet_manager.cc:1397] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:26:01.456838 8291 raft_consensus.cc:348] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.456969 8291 raft_consensus.cc:374] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:01.456991 8291 raft_consensus.cc:729] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d338d96f80d24a3cbee0bb8133c83b9d, State: Initialized, Role: FOLLOWER
I20260928 12:26:01.456997 8294 raft_consensus.cc:348] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.457106 8291 consensus_queue.cc:260] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.457178 8294 raft_consensus.cc:374] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:01.457247 8294 raft_consensus.cc:729] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d35fc052a2e4da3bb822057b8a455fd, State: Initialized, Role: FOLLOWER
I20260928 12:26:01.457396 8291 ts_tablet_manager.cc:1428] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:01.457376 8294 consensus_queue.cc:260] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.457604 8274 heartbeater.cc:503] Master 127.7.19.126:46071 was elected leader, sending a full tablet report...
I20260928 12:26:01.457639 8140 heartbeater.cc:503] Master 127.7.19.126:46071 was elected leader, sending a full tablet report...
I20260928 12:26:01.457639 8294 ts_tablet_manager.cc:1428] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 12:26:01.511799 8141 tablet.cc:2369] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:26:01.601331 8007 tablet.cc:2369] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:26:01.658921 8275 tablet.cc:2369] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:01.694438 8297 raft_consensus.cc:482] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:01.694567 8297 raft_consensus.cc:504] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.695009 8297 leader_election.cc:288] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d35fc052a2e4da3bb822057b8a455fd (127.7.19.66:39245), d338d96f80d24a3cbee0bb8133c83b9d (127.7.19.67:36963)
I20260928 12:26:01.697999 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.695356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:36925 (local address 127.7.19.67:36963)
0928 12:26:01.695397 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.695399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.695468 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:26:01.695523 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.695524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.695535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.695588 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:01.695750 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.696221 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.697262 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.697649 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.697729 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.697739 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.697901 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.697940 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:01.697999 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.695292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36963 (local address 127.7.19.65:36925)
0928 12:26:01.695377 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.695399 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.695465 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.695600 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.695600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.695602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:01.695681 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.695682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.696303 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.696304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.697165 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.697167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.697731 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.697732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.697815 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.697818 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.697876 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.697918 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":2,"thread_start_us":52,"threads_started":1}
I20260928 12:26:01.698395 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ea3d9d6c1649c0bdb9f242acac531a" candidate_uuid: "135d121d4e4641468342b70f2cc254c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" is_pre_election: true
I20260928 12:26:01.698518 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.695187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39245 (local address 127.7.19.65:42211)
0928 12:26:01.695342 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:01.695356 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:01.695454 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:01.695604 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:01.695605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:01.695607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:01.696094 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.696096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.696772 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.696773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.697476 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:01.697478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.698022 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.698023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:01.698321 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.698324 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:01.698401 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:26:01.698436 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260928 12:26:01.698509 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.695260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:42211 (local address 127.7.19.66:39245)
0928 12:26:01.695321 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.695324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.695444 (+ 120us) server_negotiation.cc:407] Connection header received
0928 12:26:01.695521 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.695522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.695539 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.695596 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:01.696173 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.696631 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.697566 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.697941 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.698020 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.698029 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.698415 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.698451 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:26:01.698560 8228 raft_consensus.cc:2454] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135d121d4e4641468342b70f2cc254c0 in term 0.
I20260928 12:26:01.698786 7894 leader_election.cc:302] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135d121d4e4641468342b70f2cc254c0, d338d96f80d24a3cbee0bb8133c83b9d; no voters:
I20260928 12:26:01.698797 8094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ea3d9d6c1649c0bdb9f242acac531a" candidate_uuid: "135d121d4e4641468342b70f2cc254c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d35fc052a2e4da3bb822057b8a455fd" is_pre_election: true
I20260928 12:26:01.698925 8094 raft_consensus.cc:2454] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135d121d4e4641468342b70f2cc254c0 in term 0.
I20260928 12:26:01.698933 8297 raft_consensus.cc:2781] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:01.698977 8297 raft_consensus.cc:482] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:01.698998 8297 raft_consensus.cc:3037] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:01.699747 8297 raft_consensus.cc:504] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.699889 8297 leader_election.cc:288] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [CANDIDATE]: Term 1 election: Requested vote from peers 9d35fc052a2e4da3bb822057b8a455fd (127.7.19.66:39245), d338d96f80d24a3cbee0bb8133c83b9d (127.7.19.67:36963)
I20260928 12:26:01.700048 8094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ea3d9d6c1649c0bdb9f242acac531a" candidate_uuid: "135d121d4e4641468342b70f2cc254c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d35fc052a2e4da3bb822057b8a455fd"
I20260928 12:26:01.700083 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ea3d9d6c1649c0bdb9f242acac531a" candidate_uuid: "135d121d4e4641468342b70f2cc254c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d338d96f80d24a3cbee0bb8133c83b9d"
I20260928 12:26:01.700115 8094 raft_consensus.cc:3037] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:01.700184 8228 raft_consensus.cc:3037] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:01.700910 8094 raft_consensus.cc:2454] T a9ea3d9d6c1649c0bdb9f242acac531a P 9d35fc052a2e4da3bb822057b8a455fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135d121d4e4641468342b70f2cc254c0 in term 1.
I20260928 12:26:01.700939 8228 raft_consensus.cc:2454] T a9ea3d9d6c1649c0bdb9f242acac531a P d338d96f80d24a3cbee0bb8133c83b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135d121d4e4641468342b70f2cc254c0 in term 1.
I20260928 12:26:01.701079 7896 leader_election.cc:302] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135d121d4e4641468342b70f2cc254c0, 9d35fc052a2e4da3bb822057b8a455fd; no voters:
I20260928 12:26:01.701200 8297 raft_consensus.cc:2781] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:01.701344 8297 raft_consensus.cc:686] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [term 1 LEADER]: Becoming Leader. State: Replica: 135d121d4e4641468342b70f2cc254c0, State: Running, Role: LEADER
I20260928 12:26:01.701435 8297 consensus_queue.cc:237] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } }
I20260928 12:26:01.702261 7815 catalog_manager.cc:5495] T a9ea3d9d6c1649c0bdb9f242acac531a P 135d121d4e4641468342b70f2cc254c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 135d121d4e4641468342b70f2cc254c0 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "135d121d4e4641468342b70f2cc254c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d35fc052a2e4da3bb822057b8a455fd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d338d96f80d24a3cbee0bb8133c83b9d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "135d121d4e4641468342b70f2cc254c0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39809 } health_report { overall_health: HEALTHY } } }
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:01.731380 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.723596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.7.19.65:39809)
0928 12:26:01.723657 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.723659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.723765 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:26:01.723845 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.723846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.723863 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.723925 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.724516 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.725062 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.725790 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.725987 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.726059 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.728336 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.728349 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.728349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.728356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.730289 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.730630 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.730634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.730634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.730687 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.730889 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.730890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.730890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.731075 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.731199 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.731240 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.731298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:01.753670 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.747011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.7.19.65:39809)
0928 12:26:01.747083 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.747084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.747135 (+ 51us) server_negotiation.cc:407] Connection header received
0928 12:26:01.747196 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.747197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.747205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.747255 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.747801 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.748272 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.749094 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.749286 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.749341 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.751478 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.751479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.751480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.751485 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.752659 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.753028 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.753028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.753029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.753076 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.753233 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.753234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.753234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.753315 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.753430 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.753521 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.753578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:01.775135 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:01.768200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.7.19.65:39809)
0928 12:26:01.768250 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:01.768253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:01.768359 (+ 106us) server_negotiation.cc:407] Connection header received
0928 12:26:01.768421 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:01.768422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:01.768430 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:01.768506 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:01.769106 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.769594 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.770469 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:01.770640 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:01.770688 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:01.773053 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:01.773054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:01.773054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:01.773060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:01.774235 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.774533 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.774534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.774534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.774581 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:01.774736 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:01.774737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:01.774737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:01.774818 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:01.774921 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:01.774988 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 12:26:01.775044 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 12:26:01.775616 7920 tablet_service.cc:2908] Rejecting scan request for tablet a9ea3d9d6c1649c0bdb9f242acac531a: Uninitialized: clean time has not yet been initialized
I20260928 12:26:01.776537 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 7877
I20260928 12:26:01.781903 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8011
I20260928 12:26:01.786995 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8145
I20260928 12:26:01.791555 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 7783
2026-09-28T12:26:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2170 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[8316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[8316](info): set up 2 sockets
Sep 28 12:26:01 dist-test-slave-cb7x krb5kdc[8316](info): commencing operation
krb5kdc: starting...
W20260928 12:26:02.840725 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:02 dist-test-slave-cb7x krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:02.959529 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41589
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:41589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:03.038293 8329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:03.038512 8329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:03.038555 8329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:03.039680 8329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:03.039743 8329 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:26:03.039765 8329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:03.039786 8329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:03.039816 8329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:03.039839 8329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:03.041309 8329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:41589
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:03.041566 8329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:03.041777 8329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:03.046490 8329 server_base.cc:1034] running on GCE node
W20260928 12:26:03.046434 8335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:03.046435 8334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:03.046468 8337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:03.046897 8329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:03.047204 8329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:03.048375 8329 hybrid_clock.cc:648] HybridClock initialized: now 1790598363048361 us; error 27 us; skew 500 ppm
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:03.055876 8329 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:03.056318 8329 webserver.cc:460] Webserver started at http://127.7.19.126:39243/ using document root <none> and password file <none>
I20260928 12:26:03.056499 8329 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:03.056537 8329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:03.056625 8329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:03.057672 8329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "a1d655c66f4a4a33a732fae878aceec0"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "545b584b0af3fa7703b46329ae1e9e32"
server_key_iv: "ca8246120369bee475c7e99fbed2ab6d"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.057968 8329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "a1d655c66f4a4a33a732fae878aceec0"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "545b584b0af3fa7703b46329ae1e9e32"
server_key_iv: "ca8246120369bee475c7e99fbed2ab6d"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.059193 8329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:03.059903 8344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.060070 8329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.060155 8329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "a1d655c66f4a4a33a732fae878aceec0"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "545b584b0af3fa7703b46329ae1e9e32"
server_key_iv: "ca8246120369bee475c7e99fbed2ab6d"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.060209 8329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:03.069469 8329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:03.071111 8329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:03.071213 8329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:03.075506 8329 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:41589
I20260928 12:26:03.075634 8406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:41589 every 8 connection(s)
I20260928 12:26:03.075847 8329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:03.076490 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.078694 8407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: Bootstrap starting.
I20260928 12:26:03.079264 8407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.079550 8407 log.cc:824] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:03.080204 8407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: No bootstrap required, opened a new log
I20260928 12:26:03.082312 8407 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d655c66f4a4a33a732fae878aceec0" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41589 } }
I20260928 12:26:03.082472 8407 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.082518 8407 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d655c66f4a4a33a732fae878aceec0, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.082675 8407 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d655c66f4a4a33a732fae878aceec0" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41589 } }
I20260928 12:26:03.082753 8407 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:03.082792 8407 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:03.082842 8407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.083338 8407 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d655c66f4a4a33a732fae878aceec0" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41589 } }
I20260928 12:26:03.083453 8407 leader_election.cc:302] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1d655c66f4a4a33a732fae878aceec0; no voters:
I20260928 12:26:03.083645 8407 leader_election.cc:288] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:03.083679 8410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:03.083676 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8329
I20260928 12:26:03.083793 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:03.083796 8410 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [term 1 LEADER]: Becoming Leader. State: Replica: a1d655c66f4a4a33a732fae878aceec0, State: Running, Role: LEADER
I20260928 12:26:03.083905 8407 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:03.083877 8410 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d655c66f4a4a33a732fae878aceec0" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41589 } }
I20260928 12:26:03.083984 7245 external_mini_cluster.cc:1428] Setting key 7e71726120d9d05d299e49038434b418
I20260928 12:26:03.084223 8410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1d655c66f4a4a33a732fae878aceec0. Latest consensus state: current_term: 1 leader_uuid: "a1d655c66f4a4a33a732fae878aceec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d655c66f4a4a33a732fae878aceec0" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41589 } } }
I20260928 12:26:03.084312 8410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:03.084237 8411 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1d655c66f4a4a33a732fae878aceec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d655c66f4a4a33a732fae878aceec0" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41589 } } }
I20260928 12:26:03.084506 8411 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:03.085170 8416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:03.085774 8416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:03.087393 8416 catalog_manager.cc:1347] Generated new cluster ID: 2b45a188be9247e284afb31c180d9f93
I20260928 12:26:03.087451 8416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:03.096325 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.084641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.7.19.126:41589)
0928 12:26:03.085211 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.085221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.085308 (+ 87us) server_negotiation.cc:407] Connection header received
0928 12:26:03.085430 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.085438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.085516 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.085669 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.086549 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.088013 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.089091 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.089733 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.089838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.092627 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.092633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.092633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.092645 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.094639 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.095047 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.095050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.095050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.095156 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.095372 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.095373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.095374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.095674 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.095842 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.095992 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.096126 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":100,"threads_started":1}
I20260928 12:26:03.104174 8416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:03.104898 8416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:03.113202 8416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: Generated new TSK 0
I20260928 12:26:03.113390 8416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:03.143410 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41589
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:03.224413 8433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:03.224619 8433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:03.224642 8433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:03.225870 8433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:03.225914 8433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:03.225935 8433 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:03.225953 8433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:03.227453 8433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:41589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:03.227682 8433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:03.227937 8433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:03.232933 8438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:03.232903 8441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:03.232949 8433 server_base.cc:1034] running on GCE node
W20260928 12:26:03.232915 8439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:03.233387 8433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:03.233578 8433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:03.234759 8433 hybrid_clock.cc:648] HybridClock initialized: now 1790598363234746 us; error 42 us; skew 500 ppm
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:03.237484 8433 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:03.237950 8433 webserver.cc:460] Webserver started at http://127.7.19.65:43595/ using document root <none> and password file <none>
I20260928 12:26:03.238173 8433 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:03.238219 8433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:03.238350 8433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:03.239421 8433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "ed62fb3deaab4d5fb7a4039a536fa46e"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "f5584428930971a3fcd1d2263d61e72f"
server_key_iv: "3b8f767e36c674d3d09302c7f4ef29aa"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.239782 8433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "ed62fb3deaab4d5fb7a4039a536fa46e"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "f5584428930971a3fcd1d2263d61e72f"
server_key_iv: "3b8f767e36c674d3d09302c7f4ef29aa"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.241163 8433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:26:03.241925 8448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.242112 8433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.242200 8433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "ed62fb3deaab4d5fb7a4039a536fa46e"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "f5584428930971a3fcd1d2263d61e72f"
server_key_iv: "3b8f767e36c674d3d09302c7f4ef29aa"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.242270 8433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:03.252584 8433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:03.254871 8433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:03.255024 8433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:03.255564 8433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:03.255626 8433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.255662 8433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:03.255697 8433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:03.262982 8433 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:46389
I20260928 12:26:03.263074 8563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:46389 every 8 connection(s)
I20260928 12:26:03.263414 8433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:03.265858 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.255987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:55393 (local address 127.7.19.126:41589)
0928 12:26:03.256059 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.256062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.256279 (+ 217us) server_negotiation.cc:407] Connection header received
0928 12:26:03.256898 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.256900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.256913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.257001 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.257730 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.258644 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.259749 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.259979 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.260067 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.263153 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.263155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.263155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.263166 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.264589 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.265007 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.265008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.265009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.265060 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.265292 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.265293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.265293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.265426 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.265489 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.265718 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.265787 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:03.266000 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.255982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.7.19.65:55393)
0928 12:26:03.256151 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.256191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.256793 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.257060 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.257063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.257365 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.257562 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.257567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.258765 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.258766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.259634 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.259636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.260092 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.260094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.260216 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.260763 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.260771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.263039 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.264662 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.264707 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.264709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.264938 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.265108 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.265109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.265109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.265196 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.265489 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.265493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.265608 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.265683 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.265801 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 12:26:03.266638 8564 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41589
I20260928 12:26:03.266721 8564 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:03.266891 8564 heartbeater.cc:511] Master 127.7.19.126:41589 requested a full tablet report, sending...
I20260928 12:26:03.267428 8361 ts_manager.cc:194] Registered new tserver with Master: ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389)
I20260928 12:26:03.267964 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8433
I20260928 12:26:03.268067 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:03.268154 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:55393
I20260928 12:26:03.268262 7245 external_mini_cluster.cc:1428] Setting key df726e02b9235b89d6fbf80c174bcd05
I20260928 12:26:03.273383 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.267609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.7.19.65:54827)
0928 12:26:03.267801 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.267813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.267902 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.268102 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.268103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.268305 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.268386 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.268389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.269188 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.269189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.269717 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.269718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.270097 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.270098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.270175 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.270640 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.270644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.270943 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.272399 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.272400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.272400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.272570 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.272795 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.272796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.272797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.272854 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.273146 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.273147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.273192 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.273241 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.273316 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260928 12:26:03.273563 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.267663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:54827 (local address 127.7.19.126:41589)
0928 12:26:03.267707 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.267709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.267875 (+ 166us) server_negotiation.cc:407] Connection header received
0928 12:26:03.267993 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.267994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.268006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.268037 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.268488 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.269099 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.269813 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.270016 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.270096 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.271030 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.271031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.271032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.271039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.272296 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.272648 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.272649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.272650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.272739 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.272923 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.272924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.272924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.273072 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.273143 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.273291 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.273433 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:03.295246 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41589
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:03.380584 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:03.380832 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:03.380858 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:03.381973 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:03.382009 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:03.382028 8573 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:03.382046 8573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:03.383366 8573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:41589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:03.383561 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:03.383780 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:03.388499 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:03.388542 8573 server_base.cc:1034] running on GCE node
W20260928 12:26:03.388726 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:03.388859 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:03.389053 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:03.389312 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:03.390466 8573 hybrid_clock.cc:648] HybridClock initialized: now 1790598363390451 us; error 32 us; skew 500 ppm
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:03.392907 8573 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:03.393343 8573 webserver.cc:460] Webserver started at http://127.7.19.66:42163/ using document root <none> and password file <none>
I20260928 12:26:03.393529 8573 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:03.393568 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:03.393652 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:03.394615 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "aed7c9ce27d44223a30f45dfc01ecb60"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "f751789fa45a1a5d746e37242c8ac183"
server_key_iv: "dc78b583f189e9893d4c6d8a0a5ae0ee"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.394917 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "aed7c9ce27d44223a30f45dfc01ecb60"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "f751789fa45a1a5d746e37242c8ac183"
server_key_iv: "dc78b583f189e9893d4c6d8a0a5ae0ee"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.396121 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:26:03.396759 8588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.396929 8573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.397019 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "aed7c9ce27d44223a30f45dfc01ecb60"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "f751789fa45a1a5d746e37242c8ac183"
server_key_iv: "dc78b583f189e9893d4c6d8a0a5ae0ee"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.397101 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:03.419294 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:03.421360 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:03.421541 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:03.422114 8573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:03.422170 8573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.422210 8573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:03.422233 8573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:03.429644 8573 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:36695
I20260928 12:26:03.429744 8703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:36695 every 8 connection(s)
I20260928 12:26:03.430074 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:03.432415 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.422531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:46943 (local address 127.7.19.126:41589)
0928 12:26:03.422612 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.422614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.422836 (+ 222us) server_negotiation.cc:407] Connection header received
0928 12:26:03.423581 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.423583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.423609 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.423694 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.424620 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.425145 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.426848 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.427044 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.427128 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.429651 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.429652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.429653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.429660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.431125 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.431510 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.431512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.431512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.431574 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.431809 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.431810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.431810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.431923 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.431997 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.432248 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.432325 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:26:03.432593 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.422499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.7.19.66:46943)
0928 12:26:03.422683 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.422734 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.423471 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.423780 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.423786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.424190 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.424523 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.424532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.425216 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.425217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.426740 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.426743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.427150 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.427151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.427324 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.427855 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.427868 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.429563 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.431217 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.431222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.431224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.431436 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.431636 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.431637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.431637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.431746 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.431987 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.431993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.432125 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.432227 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.432378 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260928 12:26:03.433087 8704 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41589
I20260928 12:26:03.433164 8704 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:03.433351 8704 heartbeater.cc:511] Master 127.7.19.126:41589 requested a full tablet report, sending...
I20260928 12:26:03.433714 8360 ts_manager.cc:194] Registered new tserver with Master: aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695)
I20260928 12:26:03.434228 8360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:46943
I20260928 12:26:03.439313 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.434136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:58819 (local address 127.7.19.126:41589)
0928 12:26:03.434170 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.434172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.434266 (+ 94us) server_negotiation.cc:407] Connection header received
0928 12:26:03.434395 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.434396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.434406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.434458 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.434746 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.435203 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.435881 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.436091 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.436162 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.437097 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.437099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.437099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.437106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.438210 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.438534 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.438535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.438535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.438595 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.438794 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.438795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.438796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.438935 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.439005 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.439158 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.439229 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:26:03.439348 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.434090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.7.19.66:58819)
0928 12:26:03.434206 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.434215 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.434320 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.434467 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.434468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.434622 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.434668 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.434669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.435305 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.435305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.435802 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.435803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.436161 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.436162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.436235 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.436690 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.436695 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.437015 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.438303 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.438304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.438304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.438465 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.438666 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.438667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.438667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.438724 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.439008 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.439009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.439053 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.439133 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.439231 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260928 12:26:03.440119 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8573
I20260928 12:26:03.440215 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:03.440420 7245 external_mini_cluster.cc:1428] Setting key dd7b52b58e7030775e441d0e06a0eba9
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:03.463549 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41589
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:03.543136 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:03.543336 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:03.543359 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:03.544456 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:03.544492 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:03.544512 8713 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:03.544529 8713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:03.546002 8713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:41589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:03.546178 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:03.546396 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:03.551383 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:03.551411 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:03.551437 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:03.551481 8713 server_base.cc:1034] running on GCE node
I20260928 12:26:03.551775 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:03.551963 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:03.553117 8713 hybrid_clock.cc:648] HybridClock initialized: now 1790598363553100 us; error 32 us; skew 500 ppm
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:03.555601 8713 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:03.556025 8713 webserver.cc:460] Webserver started at http://127.7.19.67:37299/ using document root <none> and password file <none>
I20260928 12:26:03.556244 8713 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:03.556309 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:03.556450 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:03.557602 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "290b5a50004f49cda63a547795ada48a"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "eae3ca313b5806b64d2e468d0085e150"
server_key_iv: "64339d0bda01cbfdc0317e42c2d03faf"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.557966 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "290b5a50004f49cda63a547795ada48a"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "eae3ca313b5806b64d2e468d0085e150"
server_key_iv: "64339d0bda01cbfdc0317e42c2d03faf"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.559239 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:03.559978 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.560145 8713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.560237 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "290b5a50004f49cda63a547795ada48a"
format_stamp: "Formatted at 2026-09-28 12:26:03 on dist-test-slave-cb7x"
server_key: "eae3ca313b5806b64d2e468d0085e150"
server_key_iv: "64339d0bda01cbfdc0317e42c2d03faf"
server_key_version: "encryptionkey@0"
I20260928 12:26:03.560350 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:03.589337 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:03.591728 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:03.591895 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:03.592413 8713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:03.592473 8713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.592519 8713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:03.592543 8713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:03.599650 8713 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:34921
I20260928 12:26:03.599720 8843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:34921 every 8 connection(s)
I20260928 12:26:03.600042 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:03.602679 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.592945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:49789 (local address 127.7.19.126:41589)
0928 12:26:03.593011 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.593014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.593291 (+ 277us) server_negotiation.cc:407] Connection header received
0928 12:26:03.594027 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.594028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.594038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.594111 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.595006 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.595824 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.597145 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.597346 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.597429 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.599598 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.599599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.599600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.599607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.601289 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.601750 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.601751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.601752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.601806 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.602060 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.602061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.602061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.602178 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.602250 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.602515 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.602591 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:03.602869 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.592951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.7.19.67:49789)
0928 12:26:03.593147 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.593197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.593919 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.594194 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.594198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.594588 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.594905 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.594914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.595941 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.595942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.597060 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.597063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.597451 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.597453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.597577 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.598036 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.598044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.599462 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.601414 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.601421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.601423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.601662 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.601884 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.601885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.601885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.601996 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.602240 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.602245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.602383 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.602481 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.602635 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260928 12:26:03.603371 8844 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41589
I20260928 12:26:03.603472 8844 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:03.603646 8844 heartbeater.cc:511] Master 127.7.19.126:41589 requested a full tablet report, sending...
I20260928 12:26:03.604001 8360 ts_manager.cc:194] Registered new tserver with Master: 290b5a50004f49cda63a547795ada48a (127.7.19.67:34921)
I20260928 12:26:03.604480 8360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:49789
I20260928 12:26:03.608004 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8713
I20260928 12:26:03.608098 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:03.608263 7245 external_mini_cluster.cc:1428] Setting key c0c9e01b11722c9c67046ca72aafcb7a
I20260928 12:26:03.609179 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:26:03.611159 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.604387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:42543 (local address 127.7.19.126:41589)
0928 12:26:03.604420 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.604422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.604524 (+ 102us) server_negotiation.cc:407] Connection header received
0928 12:26:03.604552 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.604553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.604562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.604619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.604976 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.605565 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.606186 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.606389 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.606475 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.607479 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.607480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.607480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.607486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.610086 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.610423 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.610424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.610425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.610476 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.610670 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.610671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.610672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.610785 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.610858 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.611001 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.611071 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:26:03.611177 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.604326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.7.19.67:42543)
0928 12:26:03.604450 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.604461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.604517 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.604658 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.604659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.604852 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.604905 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.604906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.605644 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.605645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.606124 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.606125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.606515 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.606517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.606609 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.607042 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.607047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.607400 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.610173 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.610174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.610175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.610344 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.610545 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.610546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.610546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.610603 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.610850 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.610851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.610895 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.610976 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.611074 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:03.642541 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.634266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.7.19.126:41589)
0928 12:26:03.634349 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.634352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.634362 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:03.634410 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.634411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.634422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.634521 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.635416 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.636040 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.637228 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.637417 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.637472 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.639975 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.639977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.639977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.639984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.641598 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.641888 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.641888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.641889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.641934 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.642074 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.642074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.642075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.642170 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.642242 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.642338 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.642399 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:26:03.644208 8360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:03.646087 8360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:03.652518 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.648499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.7.19.67:34921)
0928 12:26:03.648694 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.648698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.648862 (+ 164us) server_negotiation.cc:407] Connection header received
0928 12:26:03.648912 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.648914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.648930 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.649001 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.649195 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.650438 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.651592 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.652079 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.652163 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.652223 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.652370 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.652412 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260928 12:26:03.652518 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.648263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34921 (local address 127.0.0.1:45088)
0928 12:26:03.648696 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.648707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.648771 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.649064 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.649065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.649067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.649107 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.649109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.650543 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.650544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.651508 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.651509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.652161 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.652162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.652238 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.652252 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.652350 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.652402 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":28,"thread_start_us":97,"threads_started":1}
I20260928 12:26:03.652756 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.648507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.7.19.65:46389)
0928 12:26:03.648774 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.648778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.648917 (+ 139us) server_negotiation.cc:407] Connection header received
0928 12:26:03.648988 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.648990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.649007 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.649078 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.649273 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.649947 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.651258 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.652085 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.652168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.652224 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.652562 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.652609 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":87,"threads_started":1}
I20260928 12:26:03.652838 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.648285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:46389 (local address 127.0.0.1:47054)
0928 12:26:03.648860 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.648867 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.648920 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.649147 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.649148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.649150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.649195 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.649197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.650039 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.650040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.651174 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.651176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.652261 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.652262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.652328 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.652334 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.652390 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.652434 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":86,"threads_started":1}
I20260928 12:26:03.653086 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.648260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:36695 (local address 127.0.0.1:36120)
0928 12:26:03.648556 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.648613 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.648741 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.649028 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.649031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.649033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.649107 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.649112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.650557 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.650558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.651684 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.651686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.652807 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.652808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.652892 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.652896 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.652979 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.653020 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":47,"threads_started":1}
I20260928 12:26:03.653324 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.648347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.7.19.66:36695)
0928 12:26:03.648509 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.648515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.648744 (+ 229us) server_negotiation.cc:407] Connection header received
0928 12:26:03.648841 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.648845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.648863 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.648947 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.649224 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.650454 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.651791 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.652597 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.652644 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.652737 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.653145 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.653209 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260928 12:26:03.653581 8498 tablet_service.cc:1467] Processing CreateTablet for tablet a8fbaa7c8cdf452586562efe7461cc50 (DEFAULT_TABLE table=test-table [id=707cbcf65b96412c9b1a3ea72b87e3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:03.653792 8778 tablet_service.cc:1467] Processing CreateTablet for tablet a8fbaa7c8cdf452586562efe7461cc50 (DEFAULT_TABLE table=test-table [id=707cbcf65b96412c9b1a3ea72b87e3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:03.653946 8498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fbaa7c8cdf452586562efe7461cc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.654065 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fbaa7c8cdf452586562efe7461cc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.656842 8638 tablet_service.cc:1467] Processing CreateTablet for tablet a8fbaa7c8cdf452586562efe7461cc50 (DEFAULT_TABLE table=test-table [id=707cbcf65b96412c9b1a3ea72b87e3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:03.657226 8638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fbaa7c8cdf452586562efe7461cc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.657326 8867 tablet_bootstrap.cc:492] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Bootstrap starting.
I20260928 12:26:03.658416 8867 tablet_bootstrap.cc:654] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.658772 8867 log.cc:824] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:03.659577 8867 tablet_bootstrap.cc:492] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: No bootstrap required, opened a new log
I20260928 12:26:03.659646 8867 ts_tablet_manager.cc:1397] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:03.660857 8868 tablet_bootstrap.cc:492] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Bootstrap starting.
I20260928 12:26:03.661309 8870 tablet_bootstrap.cc:492] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Bootstrap starting.
I20260928 12:26:03.661477 8868 tablet_bootstrap.cc:654] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.661825 8868 log.cc:824] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:03.662016 8870 tablet_bootstrap.cc:654] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.661974 8867 raft_consensus.cc:348] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.662174 8867 raft_consensus.cc:374] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.662227 8867 raft_consensus.cc:729] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290b5a50004f49cda63a547795ada48a, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.662366 8870 log.cc:824] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:03.662375 8867 consensus_queue.cc:260] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.662671 8867 ts_tablet_manager.cc:1428] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:26:03.662730 8868 tablet_bootstrap.cc:492] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: No bootstrap required, opened a new log
I20260928 12:26:03.662798 8868 ts_tablet_manager.cc:1397] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:03.662829 8844 heartbeater.cc:503] Master 127.7.19.126:41589 was elected leader, sending a full tablet report...
I20260928 12:26:03.663563 8870 tablet_bootstrap.cc:492] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: No bootstrap required, opened a new log
I20260928 12:26:03.663627 8870 ts_tablet_manager.cc:1397] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:03.664966 8868 raft_consensus.cc:348] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.665132 8868 raft_consensus.cc:374] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.665169 8868 raft_consensus.cc:729] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed62fb3deaab4d5fb7a4039a536fa46e, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.665277 8868 consensus_queue.cc:260] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.665548 8868 ts_tablet_manager.cc:1428] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:26:03.665597 8564 heartbeater.cc:503] Master 127.7.19.126:41589 was elected leader, sending a full tablet report...
I20260928 12:26:03.666402 8870 raft_consensus.cc:348] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.666514 8870 raft_consensus.cc:374] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.666534 8870 raft_consensus.cc:729] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed7c9ce27d44223a30f45dfc01ecb60, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.666659 8870 consensus_queue.cc:260] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.666918 8870 ts_tablet_manager.cc:1428] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:26:03.666918 8704 heartbeater.cc:503] Master 127.7.19.126:41589 was elected leader, sending a full tablet report...
W20260928 12:26:03.680615 8705 tablet.cc:2369] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:03.684628 8874 raft_consensus.cc:482] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:03.684769 8874 raft_consensus.cc:504] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.685060 8874 leader_election.cc:288] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290b5a50004f49cda63a547795ada48a (127.7.19.67:34921), aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695)
I20260928 12:26:03.687983 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.685298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:49573 (local address 127.7.19.67:34921)
0928 12:26:03.685340 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.685343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.685437 (+ 94us) server_negotiation.cc:407] Connection header received
0928 12:26:03.685506 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.685507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.685518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.685594 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.685744 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.686298 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.687259 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.687627 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.687710 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.687719 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.687861 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.687906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:26:03.688009 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.685234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34921 (local address 127.7.19.65:49573)
0928 12:26:03.685386 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.685397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.685453 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.685598 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.685599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.685600 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.685670 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.685672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.686384 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.686385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.687170 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.687172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.687708 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.687709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.687795 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.687799 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.687842 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.687897 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":17}
I20260928 12:26:03.688009 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.685372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51071 (local address 127.7.19.66:36695)
0928 12:26:03.685416 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.685418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.685565 (+ 147us) server_negotiation.cc:407] Connection header received
0928 12:26:03.685693 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.685694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.685705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.685754 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.685905 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.686371 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.687262 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.687629 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.687709 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.687718 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.687903 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.687944 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:03.688032 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.685234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:36695 (local address 127.7.19.65:51071)
0928 12:26:03.685496 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.685506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.685625 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.685757 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.685758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.685760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.685822 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.685823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.686457 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.686458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.687175 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.687176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.687712 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.687713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.687807 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.687810 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.687885 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.687923 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":61,"thread_start_us":152,"threads_started":1}
I20260928 12:26:03.688378 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fbaa7c8cdf452586562efe7461cc50" candidate_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290b5a50004f49cda63a547795ada48a" is_pre_election: true
I20260928 12:26:03.688493 8658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fbaa7c8cdf452586562efe7461cc50" candidate_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" is_pre_election: true
I20260928 12:26:03.688541 8798 raft_consensus.cc:2454] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed62fb3deaab4d5fb7a4039a536fa46e in term 0.
I20260928 12:26:03.688644 8658 raft_consensus.cc:2454] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed62fb3deaab4d5fb7a4039a536fa46e in term 0.
I20260928 12:26:03.688829 8451 leader_election.cc:302] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290b5a50004f49cda63a547795ada48a, ed62fb3deaab4d5fb7a4039a536fa46e; no voters:
I20260928 12:26:03.688961 8874 raft_consensus.cc:2781] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:03.689023 8874 raft_consensus.cc:482] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:03.689044 8874 raft_consensus.cc:3037] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.689975 8874 raft_consensus.cc:504] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.690171 8874 leader_election.cc:288] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [CANDIDATE]: Term 1 election: Requested vote from peers 290b5a50004f49cda63a547795ada48a (127.7.19.67:34921), aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695)
I20260928 12:26:03.690253 8658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fbaa7c8cdf452586562efe7461cc50" candidate_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed7c9ce27d44223a30f45dfc01ecb60"
I20260928 12:26:03.690296 8798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fbaa7c8cdf452586562efe7461cc50" candidate_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290b5a50004f49cda63a547795ada48a"
I20260928 12:26:03.690320 8658 raft_consensus.cc:3037] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.690410 8798 raft_consensus.cc:3037] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.691092 8798 raft_consensus.cc:2454] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed62fb3deaab4d5fb7a4039a536fa46e in term 1.
I20260928 12:26:03.691103 8658 raft_consensus.cc:2454] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed62fb3deaab4d5fb7a4039a536fa46e in term 1.
I20260928 12:26:03.691277 8450 leader_election.cc:302] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aed7c9ce27d44223a30f45dfc01ecb60, ed62fb3deaab4d5fb7a4039a536fa46e; no voters:
I20260928 12:26:03.691439 8874 raft_consensus.cc:2781] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:03.691608 8874 raft_consensus.cc:686] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 LEADER]: Becoming Leader. State: Replica: ed62fb3deaab4d5fb7a4039a536fa46e, State: Running, Role: LEADER
I20260928 12:26:03.691728 8874 consensus_queue.cc:237] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.692476 8361 catalog_manager.cc:5495] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e reported cstate change: term changed from 0 to 1, leader changed from <none> to ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65). New cstate: current_term: 1 leader_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } health_report { overall_health: HEALTHY } } }
I20260928 12:26:03.698215 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.695609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.7.19.65:46389)
0928 12:26:03.695654 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.695656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.695705 (+ 49us) server_negotiation.cc:407] Connection header received
0928 12:26:03.695792 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.695793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.695810 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.695867 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:03.696006 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.696489 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.697359 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.697560 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.697638 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.697853 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:03.697951 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:03.698035 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.698101 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.698121 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:26:03.699909 8658 raft_consensus.cc:1261] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Refusing update from remote peer ed62fb3deaab4d5fb7a4039a536fa46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:03.699910 8798 raft_consensus.cc:1261] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Refusing update from remote peer ed62fb3deaab4d5fb7a4039a536fa46e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:03.700223 8874 consensus_queue.cc:1035] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:03.700400 8877 consensus_queue.cc:1035] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:03.702612 8880 mvcc.cc:205] Tried to move back new op lower bound from 7334290897712291840 to 7334290897681752064. Current Snapshot: MvccSnapshot[applied={T|T < 7334290897712291840}]
I20260928 12:26:03.703079 8881 mvcc.cc:205] Tried to move back new op lower bound from 7334290897712291840 to 7334290897681752064. Current Snapshot: MvccSnapshot[applied={T|T < 7334290897712291840}]
I20260928 12:26:03.703313 8882 mvcc.cc:205] Tried to move back new op lower bound from 7334290897712291840 to 7334290897681752064. Current Snapshot: MvccSnapshot[applied={T|T < 7334290897712291840}]
I20260928 12:26:03.705991 8361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:03.706069 8361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:03.706808 8361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: Sending DeleteTablet for 3 replicas of tablet a8fbaa7c8cdf452586562efe7461cc50
I20260928 12:26:03.707110 8638 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fbaa7c8cdf452586562efe7461cc50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:03 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:36120
I20260928 12:26:03.707114 8498 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fbaa7c8cdf452586562efe7461cc50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:03 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:47054
I20260928 12:26:03.707129 8778 tablet_service.cc:1514] Processing DeleteTablet for tablet a8fbaa7c8cdf452586562efe7461cc50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:03 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:45088
I20260928 12:26:03.707516 8890 tablet_replica.cc:330] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: stopping tablet replica
I20260928 12:26:03.707620 8890 raft_consensus.cc:2229] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:03.707581 8361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:03.707710 8361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:03.707732 8890 raft_consensus.cc:2258] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:03.707863 8889 tablet_replica.cc:330] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: stopping tablet replica
I20260928 12:26:03.707999 8889 raft_consensus.cc:2229] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:03.708153 8889 raft_consensus.cc:2258] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:03.708307 8890 ts_tablet_manager.cc:1905] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:03.708699 8889 ts_tablet_manager.cc:1905] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:03.709259 8638 tablet_service.cc:1467] Processing CreateTablet for tablet c1224c4aa4e340a9b7ba1ff296a0a349 (DEFAULT_TABLE table=test-table [id=fcc895fab1d843e5938df09407bd9b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:03.709393 8638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1224c4aa4e340a9b7ba1ff296a0a349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.709393 8778 tablet_service.cc:1467] Processing CreateTablet for tablet c1224c4aa4e340a9b7ba1ff296a0a349 (DEFAULT_TABLE table=test-table [id=fcc895fab1d843e5938df09407bd9b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:03.709431 8498 tablet_service.cc:1467] Processing CreateTablet for tablet c1224c4aa4e340a9b7ba1ff296a0a349 (DEFAULT_TABLE table=test-table [id=fcc895fab1d843e5938df09407bd9b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:03.709494 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1224c4aa4e340a9b7ba1ff296a0a349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.709532 8498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1224c4aa4e340a9b7ba1ff296a0a349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.710707 8867 tablet_bootstrap.cc:492] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Bootstrap starting.
I20260928 12:26:03.711051 8870 tablet_bootstrap.cc:492] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Bootstrap starting.
I20260928 12:26:03.711241 8867 tablet_bootstrap.cc:654] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.711498 8868 tablet_bootstrap.cc:492] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Bootstrap starting.
I20260928 12:26:03.711542 8870 tablet_bootstrap.cc:654] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.711872 8868 tablet_bootstrap.cc:654] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.712249 8870 tablet_bootstrap.cc:492] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: No bootstrap required, opened a new log
I20260928 12:26:03.712296 8870 ts_tablet_manager.cc:1397] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:03.712414 8867 tablet_bootstrap.cc:492] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: No bootstrap required, opened a new log
I20260928 12:26:03.712430 8870 raft_consensus.cc:348] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.712472 8867 ts_tablet_manager.cc:1397] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:03.712478 8868 tablet_bootstrap.cc:492] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: No bootstrap required, opened a new log
I20260928 12:26:03.712493 8870 raft_consensus.cc:374] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.712517 8870 raft_consensus.cc:729] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed7c9ce27d44223a30f45dfc01ecb60, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.712517 8868 ts_tablet_manager.cc:1397] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:03.712560 8870 consensus_queue.cc:260] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.712616 8867 raft_consensus.cc:348] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.712651 8870 ts_tablet_manager.cc:1428] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.712700 8867 raft_consensus.cc:374] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.712646 8868 raft_consensus.cc:348] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.712725 8867 raft_consensus.cc:729] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290b5a50004f49cda63a547795ada48a, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.712730 8868 raft_consensus.cc:374] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.712757 8868 raft_consensus.cc:729] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed62fb3deaab4d5fb7a4039a536fa46e, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.712771 8867 consensus_queue.cc:260] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.712795 8868 consensus_queue.cc:260] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.712894 8867 ts_tablet_manager.cc:1428] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.712894 8868 ts_tablet_manager.cc:1428] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.714325 8889 ts_tablet_manager.cc:1918] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:03.714376 8891 tablet_replica.cc:330] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: stopping tablet replica
I20260928 12:26:03.714498 8889 log.cc:1196] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/a8fbaa7c8cdf452586562efe7461cc50
I20260928 12:26:03.714504 8891 raft_consensus.cc:2229] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:03.714496 8890 ts_tablet_manager.cc:1918] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:03.714558 8890 log.cc:1196] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/a8fbaa7c8cdf452586562efe7461cc50
I20260928 12:26:03.714633 8891 raft_consensus.cc:2258] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:03.714823 8890 ts_tablet_manager.cc:1939] T a8fbaa7c8cdf452586562efe7461cc50 P 290b5a50004f49cda63a547795ada48a: Deleting consensus metadata
I20260928 12:26:03.715052 8891 ts_tablet_manager.cc:1905] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:03.715075 8889 ts_tablet_manager.cc:1939] T a8fbaa7c8cdf452586562efe7461cc50 P aed7c9ce27d44223a30f45dfc01ecb60: Deleting consensus metadata
I20260928 12:26:03.715394 8347 catalog_manager.cc:4841] TS 290b5a50004f49cda63a547795ada48a (127.7.19.67:34921): tablet a8fbaa7c8cdf452586562efe7461cc50 (table test-table [id=707cbcf65b96412c9b1a3ea72b87e3bd]) successfully deleted
I20260928 12:26:03.715682 8346 catalog_manager.cc:4841] TS aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695): tablet a8fbaa7c8cdf452586562efe7461cc50 (table test-table [id=707cbcf65b96412c9b1a3ea72b87e3bd]) successfully deleted
I20260928 12:26:03.716769 8891 ts_tablet_manager.cc:1918] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:03.716830 8891 log.cc:1196] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/a8fbaa7c8cdf452586562efe7461cc50
I20260928 12:26:03.717057 8891 ts_tablet_manager.cc:1939] T a8fbaa7c8cdf452586562efe7461cc50 P ed62fb3deaab4d5fb7a4039a536fa46e: Deleting consensus metadata
I20260928 12:26:03.717422 8348 catalog_manager.cc:4841] TS ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389): tablet a8fbaa7c8cdf452586562efe7461cc50 (table test-table [id=707cbcf65b96412c9b1a3ea72b87e3bd]) successfully deleted
W20260928 12:26:03.764145 8565 tablet.cc:2369] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:03.772027 8873 raft_consensus.cc:482] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:03.772127 8873 raft_consensus.cc:504] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.772485 8873 leader_election.cc:288] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389), aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695)
I20260928 12:26:03.776060 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.772647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:46389 (local address 127.7.19.67:52067)
0928 12:26:03.772735 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.772748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.772794 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.773035 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.773036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.773037 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.773110 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.773112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.773983 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.773984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.775187 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.775190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.775742 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.775743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.775833 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.775836 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.775918 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.775963 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":29}
I20260928 12:26:03.776090 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.772862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:52067 (local address 127.7.19.65:46389)
0928 12:26:03.772911 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.772913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.772921 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:03.772949 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.772949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.772958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.773026 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.773194 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.773881 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.775310 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.775660 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.775740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.775752 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.775975 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.776021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:03.776219 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.772776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:36539 (local address 127.7.19.66:36695)
0928 12:26:03.772825 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.772828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.772909 (+ 81us) server_negotiation.cc:407] Connection header received
0928 12:26:03.772963 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.772965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.772974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.773039 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:03.773241 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.773798 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.775290 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.775822 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.775915 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.775927 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.776091 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.776139 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:26:03.776432 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.772653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:36695 (local address 127.7.19.67:36539)
0928 12:26:03.772854 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.772863 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.772910 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.773076 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.773077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.773079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:03.773145 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.773146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.773878 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.773879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.775188 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.775190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.775942 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.775943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.776003 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.776006 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.776308 (+ 302us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.776351 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":45,"thread_start_us":69,"threads_started":1}
I20260928 12:26:03.776688 8658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1224c4aa4e340a9b7ba1ff296a0a349" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" is_pre_election: true
I20260928 12:26:03.776657 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1224c4aa4e340a9b7ba1ff296a0a349" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" is_pre_election: true
I20260928 12:26:03.776804 8658 raft_consensus.cc:2454] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 0.
I20260928 12:26:03.776803 8518 raft_consensus.cc:2454] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 0.
I20260928 12:26:03.776978 8730 leader_election.cc:302] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290b5a50004f49cda63a547795ada48a, aed7c9ce27d44223a30f45dfc01ecb60; no voters:
I20260928 12:26:03.777113 8873 raft_consensus.cc:2781] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:03.777153 8873 raft_consensus.cc:482] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:03.777182 8873 raft_consensus.cc:3037] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.778239 8873 raft_consensus.cc:504] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.778384 8873 leader_election.cc:288] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 election: Requested vote from peers ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389), aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695)
I20260928 12:26:03.778519 8658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1224c4aa4e340a9b7ba1ff296a0a349" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed7c9ce27d44223a30f45dfc01ecb60"
I20260928 12:26:03.778584 8658 raft_consensus.cc:3037] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.778585 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1224c4aa4e340a9b7ba1ff296a0a349" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e"
I20260928 12:26:03.778656 8518 raft_consensus.cc:3037] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.779129 8658 raft_consensus.cc:2454] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 1.
I20260928 12:26:03.779297 8730 leader_election.cc:302] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290b5a50004f49cda63a547795ada48a, aed7c9ce27d44223a30f45dfc01ecb60; no voters:
I20260928 12:26:03.779400 8873 raft_consensus.cc:2781] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:03.779502 8518 raft_consensus.cc:2454] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 1.
I20260928 12:26:03.779551 8873 raft_consensus.cc:686] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 1 LEADER]: Becoming Leader. State: Replica: 290b5a50004f49cda63a547795ada48a, State: Running, Role: LEADER
I20260928 12:26:03.779625 8873 consensus_queue.cc:237] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } }
I20260928 12:26:03.780230 8360 catalog_manager.cc:5495] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a reported cstate change: term changed from 0 to 1, leader changed from <none> to 290b5a50004f49cda63a547795ada48a (127.7.19.67). New cstate: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } health_report { overall_health: HEALTHY } } }
Sep 28 12:26:03 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:03.821590 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.813549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.7.19.126:41589)
0928 12:26:03.813605 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:03.813607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:03.813716 (+ 109us) server_negotiation.cc:407] Connection header received
0928 12:26:03.813823 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:03.813824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:03.813835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:03.813901 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:03.814484 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.815047 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.816206 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.816418 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.816492 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.819167 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:03.819169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:03.819169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:03.819175 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:03.820520 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.820945 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.820945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.820946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.820995 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:03.821172 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:03.821172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:03.821173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:03.821266 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:03.821314 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:03.821460 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 12:26:03.821527 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:26:03.821708 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:03.813451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41589 (local address 127.0.0.1:39436)
0928 12:26:03.813632 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:03.813644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:03.813731 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:03.813934 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:03.813935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:03.814340 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:03.814398 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.814402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.815143 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.815144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.816103 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:03.816106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:03.816504 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:03.816505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:03.816565 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:03.817477 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:03.817482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:03.819087 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:03.820592 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.820598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.820601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.820880 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.821053 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:03.821053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:03.821054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:03.821118 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:03.821318 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:03.821324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:03.821372 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:26:03.821438 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:26:03.821604 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":60,"thread_start_us":95,"threads_started":1}
I20260928 12:26:03.823748 8360 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:39436:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:26:03.823971 8360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:03.825373 8778 tablet_service.cc:1467] Processing CreateTablet for tablet 92ae2bde9f5442dcb3c044596f6a3906 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34f927bfc3d04d049a549ca895509296]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:03.825380 8638 tablet_service.cc:1467] Processing CreateTablet for tablet 92ae2bde9f5442dcb3c044596f6a3906 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34f927bfc3d04d049a549ca895509296]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:03.825521 8638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ae2bde9f5442dcb3c044596f6a3906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.825528 8778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ae2bde9f5442dcb3c044596f6a3906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.826589 8867 tablet_bootstrap.cc:492] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: Bootstrap starting.
I20260928 12:26:03.826748 8498 tablet_service.cc:1467] Processing CreateTablet for tablet 92ae2bde9f5442dcb3c044596f6a3906 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34f927bfc3d04d049a549ca895509296]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:03.826875 8498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ae2bde9f5442dcb3c044596f6a3906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:03.827016 8867 tablet_bootstrap.cc:654] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.827054 8870 tablet_bootstrap.cc:492] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: Bootstrap starting.
I20260928 12:26:03.827461 8870 tablet_bootstrap.cc:654] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.827973 8867 tablet_bootstrap.cc:492] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: No bootstrap required, opened a new log
I20260928 12:26:03.828045 8867 ts_tablet_manager.cc:1397] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:03.828107 8870 tablet_bootstrap.cc:492] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: No bootstrap required, opened a new log
I20260928 12:26:03.828164 8870 ts_tablet_manager.cc:1397] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:03.828218 8867 raft_consensus.cc:348] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.828327 8867 raft_consensus.cc:374] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.828356 8867 raft_consensus.cc:729] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290b5a50004f49cda63a547795ada48a, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.828351 8870 raft_consensus.cc:348] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.828399 8867 consensus_queue.cc:260] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.828434 8870 raft_consensus.cc:374] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.828462 8870 raft_consensus.cc:729] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed7c9ce27d44223a30f45dfc01ecb60, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.828452 8868 tablet_bootstrap.cc:492] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: Bootstrap starting.
I20260928 12:26:03.828493 8867 ts_tablet_manager.cc:1428] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.828504 8870 consensus_queue.cc:260] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.828600 8870 ts_tablet_manager.cc:1428] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:03.828610 8873 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:03.828702 8873 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:03.828801 8868 tablet_bootstrap.cc:654] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:03.828897 8875 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:03.828992 8875 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:03.829469 8868 tablet_bootstrap.cc:492] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: No bootstrap required, opened a new log
I20260928 12:26:03.829537 8868 ts_tablet_manager.cc:1397] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:03.829633 8868 raft_consensus.cc:348] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.829707 8868 raft_consensus.cc:374] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:03.829730 8868 raft_consensus.cc:729] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed62fb3deaab4d5fb7a4039a536fa46e, State: Initialized, Role: FOLLOWER
I20260928 12:26:03.829782 8868 consensus_queue.cc:260] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.829869 8868 ts_tablet_manager.cc:1428] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:03.829892 8874 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:03.829983 8874 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 12:26:03.850800 8845 tablet.cc:2369] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:03.980782 8873 raft_consensus.cc:482] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:03.980890 8873 raft_consensus.cc:504] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.981046 8873 leader_election.cc:288] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695), ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389)
I20260928 12:26:03.981238 8658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ae2bde9f5442dcb3c044596f6a3906" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" is_pre_election: true
I20260928 12:26:03.981245 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ae2bde9f5442dcb3c044596f6a3906" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" is_pre_election: true
I20260928 12:26:03.981340 8658 raft_consensus.cc:2454] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 0.
I20260928 12:26:03.981349 8518 raft_consensus.cc:2454] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 0.
I20260928 12:26:03.981482 8730 leader_election.cc:302] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290b5a50004f49cda63a547795ada48a, aed7c9ce27d44223a30f45dfc01ecb60; no voters:
I20260928 12:26:03.981567 8873 raft_consensus.cc:2781] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:03.981631 8873 raft_consensus.cc:482] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:03.981664 8873 raft_consensus.cc:3037] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.982360 8873 raft_consensus.cc:504] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.982496 8873 leader_election.cc:288] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 election: Requested vote from peers aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695), ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389)
I20260928 12:26:03.982686 8518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ae2bde9f5442dcb3c044596f6a3906" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e"
I20260928 12:26:03.982707 8658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ae2bde9f5442dcb3c044596f6a3906" candidate_uuid: "290b5a50004f49cda63a547795ada48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed7c9ce27d44223a30f45dfc01ecb60"
I20260928 12:26:03.982764 8518 raft_consensus.cc:3037] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.982779 8658 raft_consensus.cc:3037] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:03.983326 8658 raft_consensus.cc:2454] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 1.
I20260928 12:26:03.983474 8518 raft_consensus.cc:2454] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290b5a50004f49cda63a547795ada48a in term 1.
I20260928 12:26:03.983534 8730 leader_election.cc:302] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290b5a50004f49cda63a547795ada48a, aed7c9ce27d44223a30f45dfc01ecb60; no voters:
I20260928 12:26:03.983618 8873 raft_consensus.cc:2781] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:03.983661 8873 raft_consensus.cc:686] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [term 1 LEADER]: Becoming Leader. State: Replica: 290b5a50004f49cda63a547795ada48a, State: Running, Role: LEADER
I20260928 12:26:03.983714 8873 consensus_queue.cc:237] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } }
I20260928 12:26:03.984053 8896 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: TxnStatusTablet state changed. Reason: New leader 290b5a50004f49cda63a547795ada48a. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:03.984126 8896 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:03.984334 8908 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:26:03.984457 8908 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:26:03.984364 8359 catalog_manager.cc:5495] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a reported cstate change: term changed from 0 to 1, leader changed from <none> to 290b5a50004f49cda63a547795ada48a (127.7.19.67). New cstate: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:04.010866 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.008333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.7.19.67:34921)
0928 12:26:04.008380 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:04.008382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:04.008390 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:04.008522 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:04.008523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:04.008548 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:04.008619 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:04.008783 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.009261 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.010050 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.010243 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.010339 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.010523 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:04.010627 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:04.010706 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:04.010762 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 12:26:04.010784 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:26:04.010874 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.008240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34921 (local address 127.0.0.1:45102)
0928 12:26:04.008336 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:04.008350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:04.008423 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:04.008624 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:04.008625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:04.008626 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:04.008707 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:04.008710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.009345 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.009346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:04.009960 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:04.009963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.010326 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.010328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:04.010405 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.010426 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:04.010691 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:04.010700 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:26:04.010756 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:26:04.010805 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 12:26:04.012130 8518 raft_consensus.cc:1261] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Refusing update from remote peer 290b5a50004f49cda63a547795ada48a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:04.012419 8658 raft_consensus.cc:1261] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Refusing update from remote peer 290b5a50004f49cda63a547795ada48a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:04.012415 8896 consensus_queue.cc:1035] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:04.012614 8896 consensus_queue.cc:1035] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:04.013180 8875 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: TxnStatusTablet state changed. Reason: New leader 290b5a50004f49cda63a547795ada48a. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:04.013267 8875 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:04.013204 8874 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: TxnStatusTablet state changed. Reason: New leader 290b5a50004f49cda63a547795ada48a. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:04.013299 8874 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:04.014122 8873 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:04.014215 8873 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:04.014539 8874 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:04.014628 8874 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P ed62fb3deaab4d5fb7a4039a536fa46e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:04.014884 8896 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:04.014976 8896 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P 290b5a50004f49cda63a547795ada48a: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:04.015161 8875 tablet_replica.cc:439] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "290b5a50004f49cda63a547795ada48a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 } } peers { permanent_uuid: "290b5a50004f49cda63a547795ada48a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34921 } } peers { permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 } } }
I20260928 12:26:04.015240 8875 tablet_replica.cc:442] T 92ae2bde9f5442dcb3c044596f6a3906 P aed7c9ce27d44223a30f45dfc01ecb60: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:04.018433 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.016093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.7.19.67:34921)
0928 12:26:04.016142 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:04.016144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:04.016150 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:26:04.016167 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:04.016167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:04.016174 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:04.016218 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:04.016346 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.016832 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.017867 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.018044 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.018089 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.018245 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:04.018268 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:04.018305 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:04.018363 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:26:04.018376 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:26:04.023083 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.020730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:53681 (local address 127.7.19.67:34921)
0928 12:26:04.020775 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:04.020776 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:04.020782 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:26:04.020799 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:04.020800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:04.020806 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:04.020843 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:04.020965 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.021366 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.022452 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.022619 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.022723 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.022875 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:04.022902 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:04.022940 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:04.023010 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 12:26:04.023025 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260928 12:26:04.023155 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.020623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34921 (local address 127.7.19.67:53681)
0928 12:26:04.020702 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:04.020713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:04.020756 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:04.020854 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:04.020855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:04.020856 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:04.020911 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:04.020912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.021426 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.021427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:04.022362 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:04.022372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.022729 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.022730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:04.022792 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.022805 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:04.022955 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:04.022957 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:04.023009 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:26:04.023048 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:26:04.025388 8658 raft_consensus.cc:1261] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Refusing update from remote peer 290b5a50004f49cda63a547795ada48a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:04.025424 8518 raft_consensus.cc:1261] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Refusing update from remote peer 290b5a50004f49cda63a547795ada48a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:04.025596 8913 consensus_queue.cc:1035] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed7c9ce27d44223a30f45dfc01ecb60" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:04.025696 8873 consensus_queue.cc:1035] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed62fb3deaab4d5fb7a4039a536fa46e" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:04.045948 8359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:04.046072 8359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:04.046656 8359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1d655c66f4a4a33a732fae878aceec0: Sending DeleteTablet for 3 replicas of tablet c1224c4aa4e340a9b7ba1ff296a0a349
I20260928 12:26:04.046908 8498 tablet_service.cc:1514] Processing DeleteTablet for tablet c1224c4aa4e340a9b7ba1ff296a0a349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:04 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:47054
I20260928 12:26:04.046919 8638 tablet_service.cc:1514] Processing DeleteTablet for tablet c1224c4aa4e340a9b7ba1ff296a0a349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:04 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:36120
I20260928 12:26:04.047025 8889 tablet_replica.cc:330] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: stopping tablet replica
I20260928 12:26:04.047036 8891 tablet_replica.cc:330] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: stopping tablet replica
I20260928 12:26:04.047044 8777 tablet_service.cc:1514] Processing DeleteTablet for tablet c1224c4aa4e340a9b7ba1ff296a0a349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:04 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:45088
I20260928 12:26:04.047071 8889 raft_consensus.cc:2229] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:04.047106 8889 raft_consensus.cc:2258] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:04.047094 8891 raft_consensus.cc:2229] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:04.047122 8890 tablet_replica.cc:330] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: stopping tablet replica
I20260928 12:26:04.047133 8891 raft_consensus.cc:2258] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:04.047163 8890 raft_consensus.cc:2229] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:04.047250 8890 raft_consensus.cc:2258] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:04.047379 8889 ts_tablet_manager.cc:1905] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:04.047540 8890 ts_tablet_manager.cc:1905] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:04.048170 8891 ts_tablet_manager.cc:1905] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:04.049479 8890 ts_tablet_manager.cc:1918] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:04.049537 8890 log.cc:1196] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/c1224c4aa4e340a9b7ba1ff296a0a349
I20260928 12:26:04.049788 8890 ts_tablet_manager.cc:1939] T c1224c4aa4e340a9b7ba1ff296a0a349 P 290b5a50004f49cda63a547795ada48a: Deleting consensus metadata
I20260928 12:26:04.050060 8891 ts_tablet_manager.cc:1918] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:04.050077 8347 catalog_manager.cc:4841] TS 290b5a50004f49cda63a547795ada48a (127.7.19.67:34921): tablet c1224c4aa4e340a9b7ba1ff296a0a349 (table test-table [id=fcc895fab1d843e5938df09407bd9b67]) successfully deleted
I20260928 12:26:04.050115 8891 log.cc:1196] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/c1224c4aa4e340a9b7ba1ff296a0a349
I20260928 12:26:04.050215 8889 ts_tablet_manager.cc:1918] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:04.050262 8889 log.cc:1196] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/c1224c4aa4e340a9b7ba1ff296a0a349
I20260928 12:26:04.050330 8891 ts_tablet_manager.cc:1939] T c1224c4aa4e340a9b7ba1ff296a0a349 P ed62fb3deaab4d5fb7a4039a536fa46e: Deleting consensus metadata
I20260928 12:26:04.050478 8889 ts_tablet_manager.cc:1939] T c1224c4aa4e340a9b7ba1ff296a0a349 P aed7c9ce27d44223a30f45dfc01ecb60: Deleting consensus metadata
I20260928 12:26:04.050755 8346 catalog_manager.cc:4841] TS aed7c9ce27d44223a30f45dfc01ecb60 (127.7.19.66:36695): tablet c1224c4aa4e340a9b7ba1ff296a0a349 (table test-table [id=fcc895fab1d843e5938df09407bd9b67]) successfully deleted
I20260928 12:26:04.051110 8348 catalog_manager.cc:4841] TS ed62fb3deaab4d5fb7a4039a536fa46e (127.7.19.65:46389): tablet c1224c4aa4e340a9b7ba1ff296a0a349 (table test-table [id=fcc895fab1d843e5938df09407bd9b67]) successfully deleted
Sep 28 12:26:04 dist-test-slave-cb7x krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:04.058717 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.048084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.7.19.65:46389)
0928 12:26:04.048125 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:04.048127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:04.049272 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 12:26:04.049300 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:04.049301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:04.049312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:04.049377 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:04.050260 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.050986 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.051796 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.052107 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.052181 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.055292 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:04.055302 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:04.055303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:04.055311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:04.057436 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:04.057888 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:04.057892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:04.057893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:04.057964 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:04.058153 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:04.058154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:04.058155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:04.058337 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:04.058498 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:04.058537 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:26:04.058620 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:26:04.058975 8558 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47068
I20260928 12:26:04.066774 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:04.059960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.7.19.126:41589)
0928 12:26:04.060006 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:04.060008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:04.060067 (+ 59us) server_negotiation.cc:407] Connection header received
0928 12:26:04.060142 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:04.060143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:04.060154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:04.060206 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:04.060709 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.061493 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.062184 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:04.062475 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:04.062570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:04.063906 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:04.063907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:04.063907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:04.063915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:04.065561 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:04.065954 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:04.065955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:04.065956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:04.066018 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:04.066219 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:04.066220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:04.066221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:04.066358 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:04.066461 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:04.066591 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:26:04.066674 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:26:04.066984 8359 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39442
I20260928 12:26:04.067704 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8433
I20260928 12:26:04.075585 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8573
I20260928 12:26:04.083518 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8713
I20260928 12:26:04.089286 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8329
2026-09-28T12:26:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2297 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:04 dist-test-slave-cb7x krb5kdc[8926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:04 dist-test-slave-cb7x krb5kdc[8926](info): set up 2 sockets
Sep 28 12:26:04 dist-test-slave-cb7x krb5kdc[8926](info): commencing operation
krb5kdc: starting...
W20260928 12:26:05.124375 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:05.239682 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:40361
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:05.320070 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:05.320310 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:05.320350 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:05.321516 8939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:05.321590 8939 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:26:05.321614 8939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:05.321645 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:05.321671 8939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:05.321695 8939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:05.323192 8939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:40361
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:05.323469 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:05.323740 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:05.328469 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.328554 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.328554 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:05.328625 8939 server_base.cc:1034] running on GCE node
I20260928 12:26:05.329034 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:05.329316 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:05.330482 8939 hybrid_clock.cc:648] HybridClock initialized: now 1790598365330464 us; error 48 us; skew 500 ppm
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:05.333097 8939 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:05.333511 8939 webserver.cc:460] Webserver started at http://127.7.19.126:45087/ using document root <none> and password file <none>
I20260928 12:26:05.333709 8939 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:05.333755 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:05.333894 8939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:05.335033 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "c198c71946284d97b5e5105ec55c6735"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "f63b44121232f5ffa826157ce13ed0da"
server_key_iv: "78fa59b81a443601d63dc6ecc10c88ab"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.335368 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "c198c71946284d97b5e5105ec55c6735"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "f63b44121232f5ffa826157ce13ed0da"
server_key_iv: "78fa59b81a443601d63dc6ecc10c88ab"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.336874 8939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:05.337644 8954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.337855 8939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:05.337967 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "c198c71946284d97b5e5105ec55c6735"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "f63b44121232f5ffa826157ce13ed0da"
server_key_iv: "78fa59b81a443601d63dc6ecc10c88ab"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.338037 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:05.347831 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:05.349803 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:05.349951 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:05.354271 8939 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:40361
I20260928 12:26:05.354318 9016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:40361 every 8 connection(s)
I20260928 12:26:05.354648 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:05.355291 9017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:05.357527 9017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Bootstrap starting.
I20260928 12:26:05.358069 9017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:05.358348 9017 log.cc:824] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:05.358999 9017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: No bootstrap required, opened a new log
I20260928 12:26:05.361081 9017 raft_consensus.cc:348] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198c71946284d97b5e5105ec55c6735" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40361 } }
I20260928 12:26:05.361212 9017 raft_consensus.cc:374] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:05.361233 9017 raft_consensus.cc:729] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c198c71946284d97b5e5105ec55c6735, State: Initialized, Role: FOLLOWER
I20260928 12:26:05.361382 9017 consensus_queue.cc:260] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198c71946284d97b5e5105ec55c6735" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40361 } }
I20260928 12:26:05.361449 9017 raft_consensus.cc:388] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:05.361469 9017 raft_consensus.cc:482] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:05.361533 9017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:05.362046 9017 raft_consensus.cc:504] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198c71946284d97b5e5105ec55c6735" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40361 } }
I20260928 12:26:05.362126 9017 leader_election.cc:302] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c198c71946284d97b5e5105ec55c6735; no voters:
I20260928 12:26:05.362291 9017 leader_election.cc:288] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:05.362332 9020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:05.362509 9020 raft_consensus.cc:686] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [term 1 LEADER]: Becoming Leader. State: Replica: c198c71946284d97b5e5105ec55c6735, State: Running, Role: LEADER
I20260928 12:26:05.362540 9017 sys_catalog.cc:564] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:05.362598 9020 consensus_queue.cc:237] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198c71946284d97b5e5105ec55c6735" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40361 } }
I20260928 12:26:05.362990 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c198c71946284d97b5e5105ec55c6735. Latest consensus state: current_term: 1 leader_uuid: "c198c71946284d97b5e5105ec55c6735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198c71946284d97b5e5105ec55c6735" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40361 } } }
I20260928 12:26:05.363080 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:05.363256 9028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:05.363508 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c198c71946284d97b5e5105ec55c6735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c198c71946284d97b5e5105ec55c6735" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40361 } } }
I20260928 12:26:05.363595 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:05.363801 9028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:05.364516 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 8939
I20260928 12:26:05.364598 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:05.364766 7245 external_mini_cluster.cc:1428] Setting key dc116e383818dfd5820c3f56cb14faf0
I20260928 12:26:05.365566 9028 catalog_manager.cc:1347] Generated new cluster ID: 05d32eb046d74c96b213eb35493b39f6
I20260928 12:26:05.365620 9028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:05.373646 9028 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:05.374575 9028 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:05.376402 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.365194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.7.19.126:40361)
0928 12:26:05.365340 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.365348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.365718 (+ 370us) server_negotiation.cc:407] Connection header received
0928 12:26:05.365859 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.365866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.365916 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.366045 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.366803 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.368166 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.369334 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.369932 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.370027 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.372780 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.372784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.372785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.372796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.374828 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.375219 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.375222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.375222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.375322 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.375527 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.375528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.375528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.375788 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.375937 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.376054 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.376189 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260928 12:26:05.379737 9028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Generated new TSK 0
I20260928 12:26:05.379894 9028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:05.407146 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40361
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:05.487504 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:05.487742 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:05.487782 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:05.488962 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:05.489025 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:05.489059 9043 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:26:05.489089 9043 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:05.489121 9043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:05.490576 9043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:05.490839 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:05.491065 9043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:05.495958 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.495949 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.496031 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:05.496069 9043 server_base.cc:1034] running on GCE node
I20260928 12:26:05.496356 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:05.496547 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:05.497687 9043 hybrid_clock.cc:648] HybridClock initialized: now 1790598365497670 us; error 37 us; skew 500 ppm
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:05.500042 9043 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:05.500499 9043 webserver.cc:460] Webserver started at http://127.7.19.65:46717/ using document root <none> and password file <none>
I20260928 12:26:05.500801 9043 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:05.500850 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:05.500988 9043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:05.502092 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "b83790ed29b1467e96954c0098e48080"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "7f693cb474210a1ea8a757d2415b8c20"
server_key_iv: "341fbe7f0599d25a82b460808854e9f9"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.502449 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "b83790ed29b1467e96954c0098e48080"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "7f693cb474210a1ea8a757d2415b8c20"
server_key_iv: "341fbe7f0599d25a82b460808854e9f9"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.503714 9043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:05.504426 9058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.504588 9043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:05.504696 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "b83790ed29b1467e96954c0098e48080"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "7f693cb474210a1ea8a757d2415b8c20"
server_key_iv: "341fbe7f0599d25a82b460808854e9f9"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.504788 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:05.525748 9043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:05.527904 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:05.528067 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:05.528606 9043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:05.528666 9043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.528719 9043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:05.528750 9043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:05.535916 9043 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:39877
I20260928 12:26:05.536036 9173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:39877 every 8 connection(s)
I20260928 12:26:05.536312 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:05.538458 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.529303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:47817 (local address 127.7.19.126:40361)
0928 12:26:05.529372 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.529374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.529571 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:26:05.530113 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.530115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.530128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.530201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.530890 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.531733 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.533024 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.533245 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.533324 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.535549 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.535550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.535550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.535557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.537177 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.537585 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.537586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.537587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.537637 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.537840 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.537841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.537841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.537959 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.538033 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.538318 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.538396 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:05.538668 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.529262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.7.19.65:47817)
0928 12:26:05.529442 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.529483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.530001 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.530313 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.530318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.530602 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:05.530788 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.530793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.531853 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.531853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.532939 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.532942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.533340 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.533341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.533470 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.533963 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:05.533978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:05.535463 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:05.537253 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.537307 (+ 54us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.537308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.537522 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.537679 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.537680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.537680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.537759 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.538062 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:05.538068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:05.538187 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.538292 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.538426 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260928 12:26:05.539234 9174 heartbeater.cc:347] Connected to a master server at 127.7.19.126:40361
I20260928 12:26:05.539316 9174 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:05.539512 9174 heartbeater.cc:511] Master 127.7.19.126:40361 requested a full tablet report, sending...
I20260928 12:26:05.540084 8970 ts_manager.cc:194] Registered new tserver with Master: b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877)
I20260928 12:26:05.541064 8970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:47817
I20260928 12:26:05.542259 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 9043
I20260928 12:26:05.542361 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:05.542538 7245 external_mini_cluster.cc:1428] Setting key 5543169e5e0b2034828d7df86b71a60a
I20260928 12:26:05.546597 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.540405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51205 (local address 127.7.19.126:40361)
0928 12:26:05.540448 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.540450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.540521 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:26:05.540598 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.540599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.540607 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.540732 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.541184 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.541770 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.542679 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.542929 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.542978 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.543927 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.543928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.543929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.543935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.545535 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.545874 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.545875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.545876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.545927 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.546115 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.546117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.546117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.546219 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.546289 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.546440 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.546511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:26:05.546619 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.540318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.7.19.65:51205)
0928 12:26:05.540463 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.540474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.540544 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.540815 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.540816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.541001 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:05.541073 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.541074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.541888 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.541889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.542582 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.542584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.542990 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.542991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.543070 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.543523 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:05.543528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:05.543818 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:05.545625 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.545626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.545626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.545787 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.545994 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.545995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.545995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.546050 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.546281 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:05.546281 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:05.546334 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.546416 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.546513 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:05.568478 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40361
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:05.649748 9183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:05.649945 9183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:05.649971 9183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:05.651074 9183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:05.651114 9183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:05.651135 9183 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:26:05.651147 9183 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:05.651186 9183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:05.652658 9183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:05.652880 9183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:05.653105 9183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:05.657967 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.658043 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.657968 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:05.658223 9183 server_base.cc:1034] running on GCE node
I20260928 12:26:05.658407 9183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:05.658598 9183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:05.659734 9183 hybrid_clock.cc:648] HybridClock initialized: now 1790598365659719 us; error 28 us; skew 500 ppm
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:05.662171 9183 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:05.662714 9183 webserver.cc:460] Webserver started at http://127.7.19.66:43303/ using document root <none> and password file <none>
I20260928 12:26:05.662948 9183 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:05.663012 9183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:05.663147 9183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:05.664323 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "91bc539e82144f559fae1e7fe82e7da9"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "4c60811c22c0f21bc249ab0395202ea2"
server_key_iv: "4a5e0f54075b6c191573761cb73e1330"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.664779 9183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "91bc539e82144f559fae1e7fe82e7da9"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "4c60811c22c0f21bc249ab0395202ea2"
server_key_iv: "4a5e0f54075b6c191573761cb73e1330"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.666085 9183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:05.666896 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.667137 9183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:05.667237 9183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "91bc539e82144f559fae1e7fe82e7da9"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "4c60811c22c0f21bc249ab0395202ea2"
server_key_iv: "4a5e0f54075b6c191573761cb73e1330"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.667327 9183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:05.680228 9183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:05.682410 9183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:05.682566 9183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:05.683142 9183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:05.683198 9183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.683240 9183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:05.683281 9183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:05.690654 9183 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:43811
I20260928 12:26:05.690730 9313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:43811 every 8 connection(s)
I20260928 12:26:05.690973 9183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:05.692950 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 9183
I20260928 12:26:05.693022 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:05.693212 7245 external_mini_cluster.cc:1428] Setting key 664aab3608ead831e8638129bf0a0488
I20260928 12:26:05.694054 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.683610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:36915 (local address 127.7.19.126:40361)
0928 12:26:05.683686 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.683689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.683979 (+ 290us) server_negotiation.cc:407] Connection header received
0928 12:26:05.684732 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.684733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.684745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.684824 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.685760 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.686335 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.688006 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.688244 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.688383 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.691000 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.691002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.691003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.691011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.692503 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.692963 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.692965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.692965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.693095 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.693421 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.693422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.693422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.693538 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.693609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.693872 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.693959 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:05.694257 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.683591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.7.19.66:36915)
0928 12:26:05.683814 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.683864 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.684570 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.684924 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.684928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.685345 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:05.685660 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.685668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.686446 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.686448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.687906 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.687910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.688406 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.688407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.688569 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.689122 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:05.689132 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:05.690887 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:05.692604 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.692610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.692612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.692890 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.693193 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.693194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.693195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.693337 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.693611 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:05.693617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:05.693747 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.693858 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.694027 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260928 12:26:05.694725 9314 heartbeater.cc:347] Connected to a master server at 127.7.19.126:40361
I20260928 12:26:05.694808 9314 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:05.694998 9314 heartbeater.cc:511] Master 127.7.19.126:40361 requested a full tablet report, sending...
I20260928 12:26:05.695386 8971 ts_manager.cc:194] Registered new tserver with Master: 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811)
I20260928 12:26:05.695770 8971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:36915
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:26:05.702558 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.696441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.7.19.66:58623)
0928 12:26:05.696562 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.696574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.696636 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.696793 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.696793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.696954 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:05.697010 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.697012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.697941 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.697942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.698431 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.698432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.698962 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.698963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.699027 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.699529 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:05.699535 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:05.699828 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:05.701598 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.701599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.701600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.701759 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.701954 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.701954 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.701954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.702005 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.702318 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:05.702319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:05.702357 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.702410 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.702491 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
I20260928 12:26:05.702615 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.696521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:58623 (local address 127.7.19.126:40361)
0928 12:26:05.696569 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.696571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.696636 (+ 65us) server_negotiation.cc:407] Connection header received
0928 12:26:05.696716 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.696717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.696728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.696791 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.697075 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.697859 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.698508 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.698851 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.698932 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.699912 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.699913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.699913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.699920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.701520 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.701832 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.701833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.701834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.701890 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.702061 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.702062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.702062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.702258 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.702329 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.702478 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.702549 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:05.720733 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40361
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:05.804379 9323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:05.804639 9323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:05.804682 9323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:05.805884 9323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:05.805956 9323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:05.805991 9323 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 12:26:05.806020 9323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:05.806051 9323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:05.807619 9323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:05.807883 9323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:05.808149 9323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:05.813199 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:05.813311 9323 server_base.cc:1034] running on GCE node
W20260928 12:26:05.813206 9328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:05.813206 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:05.813644 9323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:05.813900 9323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:05.815063 9323 hybrid_clock.cc:648] HybridClock initialized: now 1790598365815054 us; error 32 us; skew 500 ppm
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:05.817564 9323 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:05.818006 9323 webserver.cc:460] Webserver started at http://127.7.19.67:40329/ using document root <none> and password file <none>
I20260928 12:26:05.818218 9323 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:05.818279 9323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:05.818421 9323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:05.819446 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "5c78767d81ba48cabd4e855ea0bc6956"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "55b926ca67cec8baf67713f5a792af69"
server_key_iv: "f2ef4fa54ebec70f67c0c1e9b366e3a4"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.819795 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "5c78767d81ba48cabd4e855ea0bc6956"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "55b926ca67cec8baf67713f5a792af69"
server_key_iv: "f2ef4fa54ebec70f67c0c1e9b366e3a4"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.821199 9323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 12:26:05.821918 9338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.822064 9323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:05.822161 9323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "5c78767d81ba48cabd4e855ea0bc6956"
format_stamp: "Formatted at 2026-09-28 12:26:05 on dist-test-slave-cb7x"
server_key: "55b926ca67cec8baf67713f5a792af69"
server_key_iv: "f2ef4fa54ebec70f67c0c1e9b366e3a4"
server_key_version: "encryptionkey@0"
I20260928 12:26:05.822230 9323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:05.842337 9323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:05.844753 9323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:05.844919 9323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:05.845423 9323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:05.845464 9323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.845489 9323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:05.845530 9323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:05.851788 9323 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:36601
I20260928 12:26:05.851938 9453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:36601 every 8 connection(s)
I20260928 12:26:05.852202 9323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:05.855372 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.845918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:36075 (local address 127.7.19.126:40361)
0928 12:26:05.845997 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.845999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.846159 (+ 160us) server_negotiation.cc:407] Connection header received
0928 12:26:05.846638 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.846639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.846653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.846724 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.847596 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.848442 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.850010 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.850316 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.850399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.853009 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.853022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.853023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.853031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.854161 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.854619 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.854620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.854620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.854666 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.854869 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.854869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.854870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.854972 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.855020 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.855237 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.855310 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:05.855466 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 9323
I20260928 12:26:05.855552 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:05.855625 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.845859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.7.19.67:36075)
0928 12:26:05.846030 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.846070 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.846545 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.846826 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.846830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.847193 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:05.847488 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.847495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.848553 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.848554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.849930 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.849934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.850407 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.850408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.850545 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.851110 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:05.851122 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:05.852928 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:05.854235 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.854240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.854242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.854544 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.854705 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.854705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.854706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.854795 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.855024 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:05.855029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:05.855134 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.855216 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.855375 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260928 12:26:05.855775 7245 external_mini_cluster.cc:1428] Setting key 7f930ce04de4e290dc5d39df8db88543
I20260928 12:26:05.856207 9454 heartbeater.cc:347] Connected to a master server at 127.7.19.126:40361
I20260928 12:26:05.856280 9454 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:05.856463 9454 heartbeater.cc:511] Master 127.7.19.126:40361 requested a full tablet report, sending...
I20260928 12:26:05.856858 8971 ts_manager.cc:194] Registered new tserver with Master: 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:05.857239 8971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:36075
I20260928 12:26:05.862052 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.857244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:48115 (local address 127.7.19.126:40361)
0928 12:26:05.857302 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.857303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.857382 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:26:05.857453 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.857454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.857463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.857522 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.857866 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.858295 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.858964 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.859135 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.859189 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.859972 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.859973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.859973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.859979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.861271 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.861558 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.861559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.861559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.861604 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.861741 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.861741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.861741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.861817 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.861860 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.861949 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.861998 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:05.862059 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.857192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.7.19.67:48115)
0928 12:26:05.857325 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.857336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.857406 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.857553 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.857554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.857733 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:05.857787 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.857789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.858379 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.858380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.858891 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.858892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.859216 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.859217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.859280 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.859627 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:05.859631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:05.859902 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:05.861343 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.861343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.861344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.861498 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.861655 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:05.861656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:05.861656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:05.861704 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:05.861856 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:05.861857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:05.861887 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.861934 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.862001 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260928 12:26:05.867180 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:26:05 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:05.889454 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.882262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.7.19.126:40361)
0928 12:26:05.882330 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.882332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.882341 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:05.882377 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.882378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.882387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.882438 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:05.882938 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.883463 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.884314 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.884527 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.884581 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.886999 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:05.887000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:05.887000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:05.887007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:05.888334 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.888649 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.888649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.888650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.888740 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:05.888984 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:05.888985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:05.888986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:05.889104 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:05.889194 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.889294 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.889360 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:05.891095 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:05.892841 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:05.899722 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.895441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.7.19.67:36601)
0928 12:26:05.895653 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.895657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.895667 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:05.895711 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.895712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.895725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.895856 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:05.896106 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.897413 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.898665 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.899237 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.899333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.899399 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.899578 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.899619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260928 12:26:05.899713 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.895276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36601 (local address 127.0.0.1:60042)
0928 12:26:05.895427 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.895469 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.895543 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.895934 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.895938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.895941 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:05.896010 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.896012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.897503 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.897504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.898576 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.898578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.899331 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.899332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.899440 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.899455 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.899559 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.899624 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":52,"threads_started":1}
I20260928 12:26:05.899928 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.895547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.7.19.65:39877)
0928 12:26:05.896141 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.896145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.896155 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:05.896190 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.896192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.896206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.896275 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:05.896340 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.897125 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.898627 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.899367 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.899446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.899496 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.899765 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.899805 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":71,"threads_started":1}
I20260928 12:26:05.899967 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.895290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.0.0.1:50698)
0928 12:26:05.895691 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.895699 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.895745 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.896270 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.896271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.896273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:05.896300 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.896302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.897231 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.897232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.898505 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.898507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.899533 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.899535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.899610 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.899619 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.899668 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.899725 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260928 12:26:05.900336 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.895278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43811 (local address 127.0.0.1:42376)
0928 12:26:05.895679 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:05.895689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:05.895745 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:05.896056 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:05.896057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:05.896059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:05.896119 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.896121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.897566 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.897567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.898573 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:05.898576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.899535 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.899536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:05.899610 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.899617 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:26:05.899668 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:26:05.900273 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":568,"thread_start_us":91,"threads_started":1}
I20260928 12:26:05.900559 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:05.895518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.7.19.66:43811)
0928 12:26:05.895855 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:05.895859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:05.895869 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:05.895928 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:05.895930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:05.895947 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:05.896024 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:05.896201 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.897470 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.898682 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:05.899376 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:05.899441 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:05.899496 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:05.899711 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 12:26:05.899752 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":82,"threads_started":1}
I20260928 12:26:05.900528 9388 tablet_service.cc:1467] Processing CreateTablet for tablet be704db3df4f4a309bf102c2cd6f3fc9 (DEFAULT_TABLE table=test-table [id=e16ea97810c8451cbe8703ba6456af24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:05.900610 9108 tablet_service.cc:1467] Processing CreateTablet for tablet be704db3df4f4a309bf102c2cd6f3fc9 (DEFAULT_TABLE table=test-table [id=e16ea97810c8451cbe8703ba6456af24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:05.900996 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be704db3df4f4a309bf102c2cd6f3fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:05.901002 9388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be704db3df4f4a309bf102c2cd6f3fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:05.901163 9248 tablet_service.cc:1467] Processing CreateTablet for tablet be704db3df4f4a309bf102c2cd6f3fc9 (DEFAULT_TABLE table=test-table [id=e16ea97810c8451cbe8703ba6456af24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:05.901458 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be704db3df4f4a309bf102c2cd6f3fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:05.903716 9477 tablet_bootstrap.cc:492] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Bootstrap starting.
I20260928 12:26:05.904839 9477 tablet_bootstrap.cc:654] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:05.904829 9478 tablet_bootstrap.cc:492] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Bootstrap starting.
I20260928 12:26:05.905155 9479 tablet_bootstrap.cc:492] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Bootstrap starting.
I20260928 12:26:05.905308 9477 log.cc:824] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:05.905493 9478 tablet_bootstrap.cc:654] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:05.905777 9479 tablet_bootstrap.cc:654] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:05.905804 9478 log.cc:824] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:05.906136 9479 log.cc:824] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:05.906188 9477 tablet_bootstrap.cc:492] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: No bootstrap required, opened a new log
I20260928 12:26:05.906270 9477 ts_tablet_manager.cc:1397] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:26:05.906574 9478 tablet_bootstrap.cc:492] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: No bootstrap required, opened a new log
I20260928 12:26:05.906634 9478 ts_tablet_manager.cc:1397] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:05.906934 9479 tablet_bootstrap.cc:492] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: No bootstrap required, opened a new log
I20260928 12:26:05.906997 9479 ts_tablet_manager.cc:1397] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:05.908133 9478 raft_consensus.cc:348] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:05.908253 9478 raft_consensus.cc:374] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:05.908295 9478 raft_consensus.cc:729] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83790ed29b1467e96954c0098e48080, State: Initialized, Role: FOLLOWER
I20260928 12:26:05.908425 9478 consensus_queue.cc:260] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:05.908705 9478 ts_tablet_manager.cc:1428] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:26:05.908727 9174 heartbeater.cc:503] Master 127.7.19.126:40361 was elected leader, sending a full tablet report...
I20260928 12:26:05.908541 9477 raft_consensus.cc:348] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:05.908771 9477 raft_consensus.cc:374] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:05.908816 9477 raft_consensus.cc:729] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c78767d81ba48cabd4e855ea0bc6956, State: Initialized, Role: FOLLOWER
I20260928 12:26:05.908946 9477 consensus_queue.cc:260] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:05.909281 9454 heartbeater.cc:503] Master 127.7.19.126:40361 was elected leader, sending a full tablet report...
I20260928 12:26:05.909214 9479 raft_consensus.cc:348] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:05.909372 9479 raft_consensus.cc:374] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:05.909408 9479 raft_consensus.cc:729] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Initialized, Role: FOLLOWER
I20260928 12:26:05.909235 9477 ts_tablet_manager.cc:1428] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:26:05.909518 9479 consensus_queue.cc:260] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:05.909808 9314 heartbeater.cc:503] Master 127.7.19.126:40361 was elected leader, sending a full tablet report...
I20260928 12:26:05.909799 9479 ts_tablet_manager.cc:1428] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 12:26:05.941619 9315 tablet.cc:2369] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:06.033664 9484 raft_consensus.cc:482] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:06.033792 9484 raft_consensus.cc:504] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:06.034128 9484 leader_election.cc:288] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811), b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877)
I20260928 12:26:06.036898 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.034341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:40117 (local address 127.7.19.66:43811)
0928 12:26:06.034392 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.034394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.034410 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:26:06.034533 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.034534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.034543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.034618 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:06.034781 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.035237 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.036158 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.036524 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.036601 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.036611 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.036791 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.036829 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:26:06.036953 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.034437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:58327 (local address 127.7.19.65:39877)
0928 12:26:06.034485 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.034487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.034588 (+ 101us) server_negotiation.cc:407] Connection header received
0928 12:26:06.034652 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.034653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.034669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.034726 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:06.034873 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.035316 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.036216 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.036570 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.036656 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.036689 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.036840 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.036879 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:06.036962 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.034336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.7.19.67:58327)
0928 12:26:06.034513 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.034523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.034591 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.034729 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.034729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.034731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:06.034802 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.034804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.035395 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.035396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.036127 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.036130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.036658 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.036659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.036761 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.036764 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.036829 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.036867 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":135,"threads_started":1}
W20260928 12:26:06.037041 9175 tablet.cc:2369] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:06.037408 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.034299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43811 (local address 127.7.19.67:40117)
0928 12:26:06.034359 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.034373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.034423 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.034621 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.034622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.034624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:06.034694 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.034695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.035313 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.035314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.036068 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.036070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.036610 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.036611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.036718 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.036721 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.036786 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.036824 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":33}
I20260928 12:26:06.037568 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be704db3df4f4a309bf102c2cd6f3fc9" candidate_uuid: "5c78767d81ba48cabd4e855ea0bc6956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83790ed29b1467e96954c0098e48080" is_pre_election: true
I20260928 12:26:06.037731 9128 raft_consensus.cc:2454] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c78767d81ba48cabd4e855ea0bc6956 in term 0.
I20260928 12:26:06.037732 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be704db3df4f4a309bf102c2cd6f3fc9" candidate_uuid: "5c78767d81ba48cabd4e855ea0bc6956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc539e82144f559fae1e7fe82e7da9" is_pre_election: true
I20260928 12:26:06.037922 9268 raft_consensus.cc:2454] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c78767d81ba48cabd4e855ea0bc6956 in term 0.
I20260928 12:26:06.037961 9339 leader_election.cc:302] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c78767d81ba48cabd4e855ea0bc6956, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:06.038096 9484 raft_consensus.cc:2781] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:06.038177 9484 raft_consensus.cc:482] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:06.038213 9484 raft_consensus.cc:3037] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.038961 9484 raft_consensus.cc:504] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:06.039111 9484 leader_election.cc:288] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [CANDIDATE]: Term 1 election: Requested vote from peers 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811), b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877)
I20260928 12:26:06.039252 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be704db3df4f4a309bf102c2cd6f3fc9" candidate_uuid: "5c78767d81ba48cabd4e855ea0bc6956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83790ed29b1467e96954c0098e48080"
I20260928 12:26:06.039252 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be704db3df4f4a309bf102c2cd6f3fc9" candidate_uuid: "5c78767d81ba48cabd4e855ea0bc6956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc539e82144f559fae1e7fe82e7da9"
I20260928 12:26:06.039332 9268 raft_consensus.cc:3037] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.039347 9128 raft_consensus.cc:3037] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.040041 9128 raft_consensus.cc:2454] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c78767d81ba48cabd4e855ea0bc6956 in term 1.
I20260928 12:26:06.040216 9268 raft_consensus.cc:2454] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c78767d81ba48cabd4e855ea0bc6956 in term 1.
I20260928 12:26:06.040252 9339 leader_election.cc:302] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c78767d81ba48cabd4e855ea0bc6956, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:06.040323 9484 raft_consensus.cc:2781] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:06.040446 9484 raft_consensus.cc:686] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 LEADER]: Becoming Leader. State: Replica: 5c78767d81ba48cabd4e855ea0bc6956, State: Running, Role: LEADER
I20260928 12:26:06.040553 9484 consensus_queue.cc:237] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } }
I20260928 12:26:06.041435 8971 catalog_manager.cc:5495] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67). New cstate: current_term: 1 leader_uuid: "5c78767d81ba48cabd4e855ea0bc6956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:26:06 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598365, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:06.068394 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.061456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40361 (local address 127.0.0.1:45640)
0928 12:26:06.061631 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.061641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.061724 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.061878 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.061878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.062194 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:06.062243 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.062246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.062999 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.063000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.063590 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.063592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.063994 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.063995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.064054 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.064576 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:06.064581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:06.066094 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:06.067479 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:06.067485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:06.067486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:06.067682 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:06.067853 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:06.067854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:06.067854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:06.067910 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:06.068111 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:06.068116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:06.068157 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.068207 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.068298 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260928 12:26:06.068482 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.061509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.7.19.126:40361)
0928 12:26:06.061589 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.061591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.061702 (+ 111us) server_negotiation.cc:407] Connection header received
0928 12:26:06.061794 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.061795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.061806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.061874 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:06.062330 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.062902 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.063672 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.063909 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.063986 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.066177 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:06.066179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:06.066179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:06.066185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:06.067409 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:06.067744 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:06.067745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:06.067745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:06.067793 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:06.067958 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:06.067959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:06.067959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:06.068061 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:06.068111 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.068234 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.068349 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":53}
I20260928 12:26:06.070493 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:45640:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:26:06.070747 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:06.072122 9108 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea28750af9a4e4e8538b592681c8fe0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ff81403442a46fa91dc1f399ee0c7c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:06.072165 9248 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea28750af9a4e4e8538b592681c8fe0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ff81403442a46fa91dc1f399ee0c7c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:06.072173 9388 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea28750af9a4e4e8538b592681c8fe0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ff81403442a46fa91dc1f399ee0c7c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:06.072304 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea28750af9a4e4e8538b592681c8fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:06.072304 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea28750af9a4e4e8538b592681c8fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:06.072335 9388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea28750af9a4e4e8538b592681c8fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:06.073781 9477 tablet_bootstrap.cc:492] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: Bootstrap starting.
I20260928 12:26:06.073817 9479 tablet_bootstrap.cc:492] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: Bootstrap starting.
I20260928 12:26:06.073808 9478 tablet_bootstrap.cc:492] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: Bootstrap starting.
I20260928 12:26:06.074146 9478 tablet_bootstrap.cc:654] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:06.074272 9477 tablet_bootstrap.cc:654] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:06.074507 9479 tablet_bootstrap.cc:654] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:06.074707 9478 tablet_bootstrap.cc:492] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: No bootstrap required, opened a new log
I20260928 12:26:06.074759 9478 ts_tablet_manager.cc:1397] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:06.074931 9478 raft_consensus.cc:348] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.074988 9477 tablet_bootstrap.cc:492] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: No bootstrap required, opened a new log
I20260928 12:26:06.075019 9478 raft_consensus.cc:374] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:06.075031 9477 ts_tablet_manager.cc:1397] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:06.075042 9478 raft_consensus.cc:729] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83790ed29b1467e96954c0098e48080, State: Initialized, Role: FOLLOWER
I20260928 12:26:06.075086 9478 consensus_queue.cc:260] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.075202 9478 ts_tablet_manager.cc:1428] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:06.075178 9477 raft_consensus.cc:348] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.075239 9479 tablet_bootstrap.cc:492] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: No bootstrap required, opened a new log
I20260928 12:26:06.075263 9477 raft_consensus.cc:374] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:06.075286 9477 raft_consensus.cc:729] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c78767d81ba48cabd4e855ea0bc6956, State: Initialized, Role: FOLLOWER
I20260928 12:26:06.075278 9479 ts_tablet_manager.cc:1397] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:06.075333 9477 consensus_queue.cc:260] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.075348 9483 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.075410 9483 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:06.075443 9477 ts_tablet_manager.cc:1428] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:06.075431 9479 raft_consensus.cc:348] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.075505 9479 raft_consensus.cc:374] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:06.075531 9479 raft_consensus.cc:729] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Initialized, Role: FOLLOWER
I20260928 12:26:06.075492 9484 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.075572 9484 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:06.075582 9479 consensus_queue.cc:260] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.075652 9479 ts_tablet_manager.cc:1428] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:06.075760 9485 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.075827 9485 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 12:26:06.102799 9455 tablet.cc:2369] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:06.385418 9483 raft_consensus.cc:482] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:06.385535 9483 raft_consensus.cc:504] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.385908 9483 leader_election.cc:288] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:06.389051 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.386186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:52905 (local address 127.7.19.66:43811)
0928 12:26:06.386232 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.386235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.386370 (+ 135us) server_negotiation.cc:407] Connection header received
0928 12:26:06.386446 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.386447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.386459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.386528 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:06.386699 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.387245 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.388219 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.388623 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.388744 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.388756 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.388934 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.388974 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:06.389078 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.386122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43811 (local address 127.7.19.65:52905)
0928 12:26:06.386267 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.386280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.386376 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.386546 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.386547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.386549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:06.386627 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.386629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.387334 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.387335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.388123 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.388125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.388787 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.388788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.388850 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.388853 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.388928 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.388972 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260928 12:26:06.389189 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.386202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:41787 (local address 127.7.19.67:36601)
0928 12:26:06.386267 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.386269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.386414 (+ 145us) server_negotiation.cc:407] Connection header received
0928 12:26:06.386485 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.386486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.386499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.386566 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:06.386718 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.387181 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.388162 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.388520 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.388603 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.388614 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.389066 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.389108 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:26:06.389325 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea28750af9a4e4e8538b592681c8fe0" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc539e82144f559fae1e7fe82e7da9" is_pre_election: true
I20260928 12:26:06.389401 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.386145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36601 (local address 127.7.19.65:41787)
0928 12:26:06.386338 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.386346 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.386425 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.386570 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.386570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.386572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:06.386648 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.386650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.387263 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.387264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.388069 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.388071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.388927 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.388929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.388987 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.388990 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.389060 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.389100 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":63,"threads_started":1}
I20260928 12:26:06.389425 9268 raft_consensus.cc:2454] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 0.
I20260928 12:26:06.389596 9061 leader_election.cc:302] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bc539e82144f559fae1e7fe82e7da9, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:06.389753 9483 raft_consensus.cc:2781] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:06.389799 9483 raft_consensus.cc:482] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:06.389822 9483 raft_consensus.cc:3037] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.389796 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea28750af9a4e4e8538b592681c8fe0" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956" is_pre_election: true
I20260928 12:26:06.389899 9408 raft_consensus.cc:2454] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 0.
I20260928 12:26:06.390592 9483 raft_consensus.cc:504] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.390738 9483 leader_election.cc:288] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 election: Requested vote from peers 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:06.390899 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea28750af9a4e4e8538b592681c8fe0" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956"
I20260928 12:26:06.390928 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea28750af9a4e4e8538b592681c8fe0" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc539e82144f559fae1e7fe82e7da9"
I20260928 12:26:06.390986 9408 raft_consensus.cc:3037] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.390991 9268 raft_consensus.cc:3037] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.391780 9268 raft_consensus.cc:2454] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 1.
I20260928 12:26:06.391884 9408 raft_consensus.cc:2454] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 1.
I20260928 12:26:06.391964 9061 leader_election.cc:302] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bc539e82144f559fae1e7fe82e7da9, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:06.392058 9483 raft_consensus.cc:2781] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:06.392175 9483 raft_consensus.cc:686] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [term 1 LEADER]: Becoming Leader. State: Replica: b83790ed29b1467e96954c0098e48080, State: Running, Role: LEADER
I20260928 12:26:06.392321 9483 consensus_queue.cc:237] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.392802 9500 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: TxnStatusTablet state changed. Reason: New leader b83790ed29b1467e96954c0098e48080. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.392900 9500 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:06.393085 9502 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:26:06.393203 9502 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:26:06.393174 8971 catalog_manager.cc:5495] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 reported cstate change: term changed from 0 to 1, leader changed from <none> to b83790ed29b1467e96954c0098e48080 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:06.431147 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.427884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.0.0.1:50712)
0928 12:26:06.427979 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.427993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.428069 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.428281 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.428282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.428283 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:06.428336 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.428339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.429426 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.429427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.430072 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.430074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.430612 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.430613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.430668 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.430683 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.430997 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.431003 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.431032 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.431076 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 12:26:06.431147 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.428007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.7.19.65:39877)
0928 12:26:06.428083 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.428085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.428121 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:26:06.428159 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.428160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.428181 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.428254 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.428419 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.429316 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.430165 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.430498 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.430577 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.430790 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.430927 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.431014 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.431038 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.431060 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:26:06.433128 9268 raft_consensus.cc:1261] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Refusing update from remote peer b83790ed29b1467e96954c0098e48080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:06.433132 9408 raft_consensus.cc:1261] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Refusing update from remote peer b83790ed29b1467e96954c0098e48080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:06.433430 9500 consensus_queue.cc:1035] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:06.433588 9483 consensus_queue.cc:1035] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:06.434640 9484 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: TxnStatusTablet state changed. Reason: New leader b83790ed29b1467e96954c0098e48080. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.434729 9484 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:06.434768 9485 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: TxnStatusTablet state changed. Reason: New leader b83790ed29b1467e96954c0098e48080. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.434855 9485 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:06.435781 9483 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.435870 9483 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:06.435921 9483 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.435971 9483 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P b83790ed29b1467e96954c0098e48080: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:06.436115 9505 mvcc.cc:205] Tried to move back new op lower bound from 7334290908907409408 to 7334290908743454720. Current Snapshot: MvccSnapshot[applied={T|T < 7334290908907409408 or (T in {7334290908907409408})}]
I20260928 12:26:06.436226 9485 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.436312 9485 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P 91bc539e82144f559fae1e7fe82e7da9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:06.436360 9484 tablet_replica.cc:439] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } } }
I20260928 12:26:06.436425 9484 tablet_replica.cc:442] T 1ea28750af9a4e4e8538b592681c8fe0 P 5c78767d81ba48cabd4e855ea0bc6956: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:06.437462 9507 mvcc.cc:205] Tried to move back new op lower bound from 7334290908907409408 to 7334290908743454720. Current Snapshot: MvccSnapshot[applied={T|T < 7334290908907409408 or (T in {7334290908907409408})}]
I20260928 12:26:06.441213 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.438108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.7.19.67:36601)
0928 12:26:06.438147 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.438149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.438223 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:26:06.438304 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.438304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.438326 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.438388 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.438542 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.439059 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.440385 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.440636 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.440741 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.440935 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.441020 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.441078 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.441151 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.441169 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:06.446394 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.443626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.7.19.67:39521)
0928 12:26:06.443790 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.443799 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.443849 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.443974 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.443974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.443975 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:06.444018 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.444019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.444971 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.444972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.445544 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.445546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.446035 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.446036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.446082 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.446093 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.446267 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.446269 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.446302 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.446333 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 12:26:06.446406 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.443687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:39521 (local address 127.7.19.65:39877)
0928 12:26:06.443726 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.443728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.443861 (+ 133us) server_negotiation.cc:407] Connection header received
0928 12:26:06.443890 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.443891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.443902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.443972 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.444065 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.444899 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.445625 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.445964 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.446038 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.446177 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.446211 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.446268 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.446308 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.446327 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:26:06.448969 9128 raft_consensus.cc:1261] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Refusing update from remote peer 5c78767d81ba48cabd4e855ea0bc6956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:06.449210 9268 raft_consensus.cc:1261] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Refusing update from remote peer 5c78767d81ba48cabd4e855ea0bc6956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:06.449252 9487 consensus_queue.cc:1035] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:06.449383 9484 consensus_queue.cc:1035] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:06.458289 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.455112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36601 (local address 127.7.19.65:45587)
0928 12:26:06.455252 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.455263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.455311 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.455552 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.455553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.455563 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:06.455618 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.455619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.456590 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.456591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.457476 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.457478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.457959 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.457960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.458004 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.458016 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.458172 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.458173 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.458206 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.458237 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260928 12:26:06.458302 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.455398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:45587 (local address 127.7.19.67:36601)
0928 12:26:06.455440 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.455442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.455451 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:06.455472 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.455473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.455484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.455540 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.455675 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.456520 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.457550 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.457886 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.457964 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.458084 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.458118 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.458172 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.458210 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.458228 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:06.467583 8971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:06.467698 8971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:06.468494 8971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Sending DeleteTablet for 3 replicas of tablet be704db3df4f4a309bf102c2cd6f3fc9
I20260928 12:26:06.468814 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet be704db3df4f4a309bf102c2cd6f3fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:06 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:42376
I20260928 12:26:06.468936 9388 tablet_service.cc:1514] Processing DeleteTablet for tablet be704db3df4f4a309bf102c2cd6f3fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:06 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:60042
I20260928 12:26:06.469216 9108 tablet_service.cc:1514] Processing DeleteTablet for tablet be704db3df4f4a309bf102c2cd6f3fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:06 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:50698
I20260928 12:26:06.469429 9527 tablet_replica.cc:330] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: stopping tablet replica
I20260928 12:26:06.469439 8971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:26:06.469530 9527 raft_consensus.cc:2229] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 LEADER]: Raft consensus shutting down.
W20260928 12:26:06.469571 8971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:06.469623 9527 raft_consensus.cc:2258] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:06.470727 9527 ts_tablet_manager.cc:1905] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:06.470753 9108 tablet_service.cc:1467] Processing CreateTablet for tablet cdd7a8361d1249829649bacf5dc10786 (DEFAULT_TABLE table=test-table [id=16e222d759eb419d9fb22b09a93e8ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:06.470899 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd7a8361d1249829649bacf5dc10786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:06.471298 9528 tablet_replica.cc:330] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: stopping tablet replica
I20260928 12:26:06.471395 9528 raft_consensus.cc:2229] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:06.471439 9528 raft_consensus.cc:2258] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:06.472193 9478 tablet_bootstrap.cc:492] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Bootstrap starting.
I20260928 12:26:06.472549 9478 tablet_bootstrap.cc:654] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:06.473011 9248 tablet_service.cc:1467] Processing CreateTablet for tablet cdd7a8361d1249829649bacf5dc10786 (DEFAULT_TABLE table=test-table [id=16e222d759eb419d9fb22b09a93e8ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:06.473114 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd7a8361d1249829649bacf5dc10786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:06.473430 9527 ts_tablet_manager.cc:1918] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:06.473495 9527 log.cc:1196] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/be704db3df4f4a309bf102c2cd6f3fc9
I20260928 12:26:06.473789 9527 ts_tablet_manager.cc:1939] T be704db3df4f4a309bf102c2cd6f3fc9 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting consensus metadata
I20260928 12:26:06.472195 9528 ts_tablet_manager.cc:1905] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:06.474392 9479 tablet_bootstrap.cc:492] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Bootstrap starting.
I20260928 12:26:06.474756 9479 tablet_bootstrap.cc:654] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:06.475584 9528 ts_tablet_manager.cc:1918] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:06.475672 9528 log.cc:1196] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/be704db3df4f4a309bf102c2cd6f3fc9
I20260928 12:26:06.475672 9388 tablet_service.cc:1467] Processing CreateTablet for tablet cdd7a8361d1249829649bacf5dc10786 (DEFAULT_TABLE table=test-table [id=16e222d759eb419d9fb22b09a93e8ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:06.475769 9388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd7a8361d1249829649bacf5dc10786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:06.475976 9528 ts_tablet_manager.cc:1939] T be704db3df4f4a309bf102c2cd6f3fc9 P b83790ed29b1467e96954c0098e48080: Deleting consensus metadata
I20260928 12:26:06.476383 8955 catalog_manager.cc:4841] TS b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877): tablet be704db3df4f4a309bf102c2cd6f3fc9 (table test-table [id=e16ea97810c8451cbe8703ba6456af24]) successfully deleted
I20260928 12:26:06.476543 9526 tablet_replica.cc:330] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: stopping tablet replica
I20260928 12:26:06.476652 9526 raft_consensus.cc:2229] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:06.476749 9526 raft_consensus.cc:2258] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:06.477010 9478 tablet_bootstrap.cc:492] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: No bootstrap required, opened a new log
I20260928 12:26:06.477056 9478 ts_tablet_manager.cc:1397] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s
I20260928 12:26:06.477173 9526 ts_tablet_manager.cc:1905] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:06.477195 9478 raft_consensus.cc:348] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.477281 9478 raft_consensus.cc:374] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:06.477305 9478 raft_consensus.cc:729] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83790ed29b1467e96954c0098e48080, State: Initialized, Role: FOLLOWER
I20260928 12:26:06.477326 9477 tablet_bootstrap.cc:492] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Bootstrap starting.
I20260928 12:26:06.477346 9478 consensus_queue.cc:260] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.477449 9478 ts_tablet_manager.cc:1428] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:06.477702 9477 tablet_bootstrap.cc:654] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:06.478765 8956 catalog_manager.cc:4841] TS 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601): tablet be704db3df4f4a309bf102c2cd6f3fc9 (table test-table [id=e16ea97810c8451cbe8703ba6456af24]) successfully deleted
I20260928 12:26:06.479988 9526 ts_tablet_manager.cc:1918] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:06.480069 9526 log.cc:1196] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/be704db3df4f4a309bf102c2cd6f3fc9
I20260928 12:26:06.480340 9526 ts_tablet_manager.cc:1939] T be704db3df4f4a309bf102c2cd6f3fc9 P 91bc539e82144f559fae1e7fe82e7da9: Deleting consensus metadata
I20260928 12:26:06.480870 8957 catalog_manager.cc:4841] TS 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811): tablet be704db3df4f4a309bf102c2cd6f3fc9 (table test-table [id=e16ea97810c8451cbe8703ba6456af24]) successfully deleted
I20260928 12:26:06.481396 9479 tablet_bootstrap.cc:492] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: No bootstrap required, opened a new log
I20260928 12:26:06.481443 9479 ts_tablet_manager.cc:1397] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 12:26:06.481591 9479 raft_consensus.cc:348] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.481657 9479 raft_consensus.cc:374] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:06.481676 9479 raft_consensus.cc:729] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Initialized, Role: FOLLOWER
I20260928 12:26:06.481715 9479 consensus_queue.cc:260] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.481788 9479 ts_tablet_manager.cc:1428] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:06.485087 9477 tablet_bootstrap.cc:492] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: No bootstrap required, opened a new log
I20260928 12:26:06.485144 9477 ts_tablet_manager.cc:1397] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 12:26:06.485373 9477 raft_consensus.cc:348] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.485448 9477 raft_consensus.cc:374] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:06.485471 9477 raft_consensus.cc:729] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c78767d81ba48cabd4e855ea0bc6956, State: Initialized, Role: FOLLOWER
I20260928 12:26:06.485514 9477 consensus_queue.cc:260] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.485607 9477 ts_tablet_manager.cc:1428] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:06.582297 9485 raft_consensus.cc:482] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:06.582412 9485 raft_consensus.cc:504] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.582731 9485 leader_election.cc:288] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:06.586585 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.583021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:38529 (local address 127.7.19.65:39877)
0928 12:26:06.583066 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.583069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.583129 (+ 60us) server_negotiation.cc:407] Connection header received
0928 12:26:06.583225 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.583226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.583237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.583302 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:06.583599 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.584295 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.585651 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.586207 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.586289 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.586299 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.586477 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.586518 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:06.586585 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.583037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:56139 (local address 127.7.19.67:36601)
0928 12:26:06.583082 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.583085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.583214 (+ 129us) server_negotiation.cc:407] Connection header received
0928 12:26:06.583311 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.583312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.583320 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.583376 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:06.583542 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.584048 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.585651 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.586216 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.586289 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.586299 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.586478 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.586514 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:06.586609 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.582917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36601 (local address 127.7.19.66:56139)
0928 12:26:06.583138 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.583149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.583241 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.583378 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.583378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.583380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:06.583459 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.583461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.584130 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.584131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.585398 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.585409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.586299 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.586300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.586377 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.586381 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.586465 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.586493 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":18,"thread_start_us":44,"threads_started":1}
I20260928 12:26:06.586606 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.582917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.7.19.66:38529)
0928 12:26:06.583054 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.583067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.583160 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.583441 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.583442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.583443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:06.583534 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.583536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.584384 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.584385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.585430 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.585432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.586300 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.586301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.586379 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.586383 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.586444 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.586491 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260928 12:26:06.586994 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd7a8361d1249829649bacf5dc10786" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956" is_pre_election: true
I20260928 12:26:06.587004 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd7a8361d1249829649bacf5dc10786" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83790ed29b1467e96954c0098e48080" is_pre_election: true
I20260928 12:26:06.587090 9408 raft_consensus.cc:2454] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 0.
I20260928 12:26:06.587107 9128 raft_consensus.cc:2454] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 0.
I20260928 12:26:06.587270 9200 leader_election.cc:302] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c78767d81ba48cabd4e855ea0bc6956, 91bc539e82144f559fae1e7fe82e7da9; no voters:
I20260928 12:26:06.587415 9485 raft_consensus.cc:2781] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:06.587474 9485 raft_consensus.cc:482] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:06.587507 9485 raft_consensus.cc:3037] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.588181 9485 raft_consensus.cc:504] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.588336 9485 leader_election.cc:288] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 election: Requested vote from peers b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:06.588485 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd7a8361d1249829649bacf5dc10786" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956"
I20260928 12:26:06.588514 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd7a8361d1249829649bacf5dc10786" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83790ed29b1467e96954c0098e48080"
I20260928 12:26:06.588574 9408 raft_consensus.cc:3037] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.588585 9128 raft_consensus.cc:3037] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:06.589169 9408 raft_consensus.cc:2454] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 1.
I20260928 12:26:06.589326 9128 raft_consensus.cc:2454] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 1.
I20260928 12:26:06.589366 9200 leader_election.cc:302] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c78767d81ba48cabd4e855ea0bc6956, 91bc539e82144f559fae1e7fe82e7da9; no voters:
I20260928 12:26:06.589484 9485 raft_consensus.cc:2781] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:06.589625 9485 raft_consensus.cc:686] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 LEADER]: Becoming Leader. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Running, Role: LEADER
I20260928 12:26:06.589681 9485 consensus_queue.cc:237] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:06.590394 8971 catalog_manager.cc:5495] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "91bc539e82144f559fae1e7fe82e7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:06.626120 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.623626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.7.19.66:43811)
0928 12:26:06.623672 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.623674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.623691 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:26:06.623731 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.623732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.623758 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.623833 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.623939 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.624462 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.625317 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.625505 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.625596 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.625754 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.625860 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.625993 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.626016 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.626035 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:26:06.630841 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.628299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.7.19.66:56447)
0928 12:26:06.628509 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.628519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.628594 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.628768 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.628769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.628770 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:06.628839 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.628840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.629463 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.629464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.630037 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.630039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.630412 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.630414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.630474 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.630485 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.630660 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.630662 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.630714 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.630754 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260928 12:26:06.630824 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.628384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:56447 (local address 127.7.19.65:39877)
0928 12:26:06.628466 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.628469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.628580 (+ 111us) server_negotiation.cc:407] Connection header received
0928 12:26:06.628656 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.628657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.628690 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.628764 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.628911 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.629380 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.630121 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.630313 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.630393 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.630566 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.630602 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.630659 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.630720 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.630749 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 12:26:06.632862 9408 raft_consensus.cc:1261] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Refusing update from remote peer 91bc539e82144f559fae1e7fe82e7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:06.632862 9128 raft_consensus.cc:1261] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Refusing update from remote peer 91bc539e82144f559fae1e7fe82e7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:06.633134 9485 consensus_queue.cc:1035] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:06.633297 9534 consensus_queue.cc:1035] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:06.639314 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.637192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:33301 (local address 127.7.19.66:43811)
0928 12:26:06.637257 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:06.637259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:06.637266 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:26:06.637297 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:06.637297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:06.637305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:06.637349 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:06.637469 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.637940 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.638639 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.638883 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.638945 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.639085 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.639115 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.639161 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:06.639237 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 12:26:06.639251 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:26:06.639341 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:06.637142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43811 (local address 127.7.19.65:33301)
0928 12:26:06.637180 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:06.637192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:06.637239 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:06.637349 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:06.637350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:06.637351 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:06.637388 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.637389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.638017 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.638018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.638558 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:06.638560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:06.638943 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:06.638944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:06.638997 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:06.639003 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:06.639185 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:06.639186 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:26:06.639229 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 12:26:06.639272 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 12:26:06.639631 9507 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:26:08.157500 8970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:08.157615 8970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:08.158368 8970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Sending DeleteTablet for 3 replicas of tablet cdd7a8361d1249829649bacf5dc10786
I20260928 12:26:08.158656 9108 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd7a8361d1249829649bacf5dc10786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:08 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:50698
I20260928 12:26:08.158654 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd7a8361d1249829649bacf5dc10786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:08 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:42376
I20260928 12:26:08.158689 9388 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd7a8361d1249829649bacf5dc10786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:08 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:60042
I20260928 12:26:08.158972 9566 tablet_replica.cc:330] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: stopping tablet replica
I20260928 12:26:08.159051 9566 raft_consensus.cc:2229] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:08.159097 9566 raft_consensus.cc:2258] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:08.159268 9568 tablet_replica.cc:330] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: stopping tablet replica
I20260928 12:26:08.159380 9566 ts_tablet_manager.cc:1905] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:08.159430 9567 tablet_replica.cc:330] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: stopping tablet replica
I20260928 12:26:08.159534 9568 raft_consensus.cc:2229] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:08.159623 9567 raft_consensus.cc:2229] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:08.159739 9567 raft_consensus.cc:2258] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:08.159804 9568 raft_consensus.cc:2258] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:08.160041 9568 ts_tablet_manager.cc:1905] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:08.160348 9567 ts_tablet_manager.cc:1905] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:08.161402 9566 ts_tablet_manager.cc:1918] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:26:08.161470 9566 log.cc:1196] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/cdd7a8361d1249829649bacf5dc10786
I20260928 12:26:08.161705 9566 ts_tablet_manager.cc:1939] T cdd7a8361d1249829649bacf5dc10786 P b83790ed29b1467e96954c0098e48080: Deleting consensus metadata
I20260928 12:26:08.162168 8955 catalog_manager.cc:4841] TS b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877): tablet cdd7a8361d1249829649bacf5dc10786 (table test-table [id=16e222d759eb419d9fb22b09a93e8ea8]) successfully deleted
I20260928 12:26:08.163570 9567 ts_tablet_manager.cc:1918] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:26:08.163620 9567 log.cc:1196] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/cdd7a8361d1249829649bacf5dc10786
I20260928 12:26:08.163838 9567 ts_tablet_manager.cc:1939] T cdd7a8361d1249829649bacf5dc10786 P 91bc539e82144f559fae1e7fe82e7da9: Deleting consensus metadata
I20260928 12:26:08.164170 8957 catalog_manager.cc:4841] TS 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811): tablet cdd7a8361d1249829649bacf5dc10786 (table test-table [id=16e222d759eb419d9fb22b09a93e8ea8]) successfully deleted
I20260928 12:26:08.164798 9568 ts_tablet_manager.cc:1918] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:26:08.164862 9568 log.cc:1196] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/cdd7a8361d1249829649bacf5dc10786
I20260928 12:26:08.165069 9568 ts_tablet_manager.cc:1939] T cdd7a8361d1249829649bacf5dc10786 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting consensus metadata
I20260928 12:26:08.165369 8956 catalog_manager.cc:4841] TS 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601): tablet cdd7a8361d1249829649bacf5dc10786 (table test-table [id=16e222d759eb419d9fb22b09a93e8ea8]) successfully deleted
Sep 28 12:26:08 dist-test-slave-cb7x krb5kdc[8926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:26:08 dist-test-slave-cb7x krb5kdc[8926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:08.191680 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:08.177916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.7.19.126:40361)
0928 12:26:08.178060 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:08.178062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:08.178076 (+ 14us) server_negotiation.cc:407] Connection header received
0928 12:26:08.178166 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:08.178167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:08.178179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:08.178241 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:08.178802 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.179309 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.180348 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.180574 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.180730 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:08.189307 (+ 8577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:08.189308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:08.189308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:08.189315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:08.190704 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:08.191024 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:08.191024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:08.191025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:08.191079 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:08.191258 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:08.191259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:08.191259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:08.191365 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:08.191439 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:08.191544 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 12:26:08.191612 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 12:26:08.193226 8970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:08.193393 8970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:08.194758 9248 tablet_service.cc:1467] Processing CreateTablet for tablet 219836a59f6844fba385858b517e70a6 (DEFAULT_TABLE table=test-table [id=423ba13aecda49ac9e6e83dce2d1bce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:08.194777 9108 tablet_service.cc:1467] Processing CreateTablet for tablet 219836a59f6844fba385858b517e70a6 (DEFAULT_TABLE table=test-table [id=423ba13aecda49ac9e6e83dce2d1bce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:08.194782 9388 tablet_service.cc:1467] Processing CreateTablet for tablet 219836a59f6844fba385858b517e70a6 (DEFAULT_TABLE table=test-table [id=423ba13aecda49ac9e6e83dce2d1bce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:08.194897 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219836a59f6844fba385858b517e70a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:08.194905 9388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219836a59f6844fba385858b517e70a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:08.194893 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219836a59f6844fba385858b517e70a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:08.196287 9578 tablet_bootstrap.cc:492] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Bootstrap starting.
I20260928 12:26:08.196319 9579 tablet_bootstrap.cc:492] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Bootstrap starting.
I20260928 12:26:08.196476 9577 tablet_bootstrap.cc:492] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Bootstrap starting.
I20260928 12:26:08.196696 9579 tablet_bootstrap.cc:654] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:08.196671 9578 tablet_bootstrap.cc:654] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:08.196905 9577 tablet_bootstrap.cc:654] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:08.197364 9578 tablet_bootstrap.cc:492] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: No bootstrap required, opened a new log
I20260928 12:26:08.197417 9578 ts_tablet_manager.cc:1397] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:08.197507 9579 tablet_bootstrap.cc:492] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: No bootstrap required, opened a new log
I20260928 12:26:08.197559 9579 ts_tablet_manager.cc:1397] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:08.197548 9578 raft_consensus.cc:348] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.197616 9578 raft_consensus.cc:374] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:08.197640 9578 raft_consensus.cc:729] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Initialized, Role: FOLLOWER
I20260928 12:26:08.197685 9578 consensus_queue.cc:260] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.197678 9579 raft_consensus.cc:348] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.197737 9579 raft_consensus.cc:374] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:08.197762 9579 raft_consensus.cc:729] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83790ed29b1467e96954c0098e48080, State: Initialized, Role: FOLLOWER
I20260928 12:26:08.197794 9578 ts_tablet_manager.cc:1428] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:08.197963 9577 tablet_bootstrap.cc:492] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: No bootstrap required, opened a new log
I20260928 12:26:08.198015 9577 ts_tablet_manager.cc:1397] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:08.198139 9577 raft_consensus.cc:348] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.198218 9577 raft_consensus.cc:374] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:08.198238 9577 raft_consensus.cc:729] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c78767d81ba48cabd4e855ea0bc6956, State: Initialized, Role: FOLLOWER
I20260928 12:26:08.198283 9577 consensus_queue.cc:260] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.198464 9577 ts_tablet_manager.cc:1428] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:08.198576 9579 consensus_queue.cc:260] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.198704 9579 ts_tablet_manager.cc:1428] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:08.251001 9560 raft_consensus.cc:482] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:08.251101 9560 raft_consensus.cc:504] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.251286 9560 leader_election.cc:288] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:08.251542 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219836a59f6844fba385858b517e70a6" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956" is_pre_election: true
I20260928 12:26:08.251542 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219836a59f6844fba385858b517e70a6" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83790ed29b1467e96954c0098e48080" is_pre_election: true
I20260928 12:26:08.251654 9408 raft_consensus.cc:2454] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 0.
I20260928 12:26:08.251654 9128 raft_consensus.cc:2454] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 0.
I20260928 12:26:08.251844 9199 leader_election.cc:302] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bc539e82144f559fae1e7fe82e7da9, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:08.251955 9560 raft_consensus.cc:2781] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:08.251997 9560 raft_consensus.cc:482] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:08.252022 9560 raft_consensus.cc:3037] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:08.252822 9560 raft_consensus.cc:504] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.252965 9560 leader_election.cc:288] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 election: Requested vote from peers b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:08.253072 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219836a59f6844fba385858b517e70a6" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83790ed29b1467e96954c0098e48080"
I20260928 12:26:08.253139 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "219836a59f6844fba385858b517e70a6" candidate_uuid: "91bc539e82144f559fae1e7fe82e7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956"
I20260928 12:26:08.253203 9408 raft_consensus.cc:3037] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:08.253206 9128 raft_consensus.cc:3037] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:08.253969 9408 raft_consensus.cc:2454] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 1.
I20260928 12:26:08.254005 9128 raft_consensus.cc:2454] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bc539e82144f559fae1e7fe82e7da9 in term 1.
I20260928 12:26:08.254140 9200 leader_election.cc:302] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c78767d81ba48cabd4e855ea0bc6956, 91bc539e82144f559fae1e7fe82e7da9; no voters:
I20260928 12:26:08.254263 9560 raft_consensus.cc:2781] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:08.254326 9560 raft_consensus.cc:686] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 LEADER]: Becoming Leader. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Running, Role: LEADER
I20260928 12:26:08.254377 9560 consensus_queue.cc:237] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.254990 8970 catalog_manager.cc:5495] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "91bc539e82144f559fae1e7fe82e7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:08.273965 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:08.271537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.7.19.66:43811)
0928 12:26:08.271677 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:08.271679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:08.271685 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:26:08.271712 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:08.271713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:08.271722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:08.271792 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:08.271916 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.272435 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.273258 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.273472 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.273543 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:08.273685 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:08.273718 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:08.273780 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:08.273839 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:26:08.273858 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260928 12:26:08.275935 9408 raft_consensus.cc:1261] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Refusing update from remote peer 91bc539e82144f559fae1e7fe82e7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:08.275955 9128 raft_consensus.cc:1261] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Refusing update from remote peer 91bc539e82144f559fae1e7fe82e7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:08.276151 9557 consensus_queue.cc:1035] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:08.276249 9560 consensus_queue.cc:1035] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:08.277559 9548 mvcc.cc:205] Tried to move back new op lower bound from 7334290916457119744 to 7334290916370411520. Current Snapshot: MvccSnapshot[applied={T|T < 7334290916457119744}]
I20260928 12:26:08.285499 8969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:08.285607 8969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:08.286228 8969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Sending DeleteTablet for 3 replicas of tablet 219836a59f6844fba385858b517e70a6
I20260928 12:26:08.286448 9108 tablet_service.cc:1514] Processing DeleteTablet for tablet 219836a59f6844fba385858b517e70a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:08 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:50698
I20260928 12:26:08.286454 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet 219836a59f6844fba385858b517e70a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:08 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:42376
I20260928 12:26:08.286520 9388 tablet_service.cc:1514] Processing DeleteTablet for tablet 219836a59f6844fba385858b517e70a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:08 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:60042
I20260928 12:26:08.286547 9567 tablet_replica.cc:330] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: stopping tablet replica
I20260928 12:26:08.286547 9566 tablet_replica.cc:330] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: stopping tablet replica
I20260928 12:26:08.286597 9567 raft_consensus.cc:2229] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:08.286597 9566 raft_consensus.cc:2229] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:08.286636 9566 raft_consensus.cc:2258] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:08.286645 9567 raft_consensus.cc:2258] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:08.286697 9568 tablet_replica.cc:330] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: stopping tablet replica
I20260928 12:26:08.286757 9568 raft_consensus.cc:2229] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:08.286803 9568 raft_consensus.cc:2258] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:08.286937 8969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:26:08.286975 9567 ts_tablet_manager.cc:1905] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:08.287063 9566 ts_tablet_manager.cc:1905] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:26:08.287081 8969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:08.288129 9108 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7b05960b8f406186f865b29e08d69a (DEFAULT_TABLE table=test-table [id=5874155f3d3045b5a5f969cf9a60b5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:08.288159 9388 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7b05960b8f406186f865b29e08d69a (DEFAULT_TABLE table=test-table [id=5874155f3d3045b5a5f969cf9a60b5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:08.288239 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b05960b8f406186f865b29e08d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:08.288244 9388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b05960b8f406186f865b29e08d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:08.288733 9567 ts_tablet_manager.cc:1918] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:08.288787 9567 log.cc:1196] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/219836a59f6844fba385858b517e70a6
I20260928 12:26:08.288833 9568 ts_tablet_manager.cc:1905] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:08.289028 9567 ts_tablet_manager.cc:1939] T 219836a59f6844fba385858b517e70a6 P 91bc539e82144f559fae1e7fe82e7da9: Deleting consensus metadata
I20260928 12:26:08.289017 9566 ts_tablet_manager.cc:1918] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:08.289088 9566 log.cc:1196] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/219836a59f6844fba385858b517e70a6
I20260928 12:26:08.289309 9566 ts_tablet_manager.cc:1939] T 219836a59f6844fba385858b517e70a6 P b83790ed29b1467e96954c0098e48080: Deleting consensus metadata
I20260928 12:26:08.289613 8955 catalog_manager.cc:4841] TS b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877): tablet 219836a59f6844fba385858b517e70a6 (table test-table [id=423ba13aecda49ac9e6e83dce2d1bce1]) successfully deleted
I20260928 12:26:08.289740 8957 catalog_manager.cc:4841] TS 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811): tablet 219836a59f6844fba385858b517e70a6 (table test-table [id=423ba13aecda49ac9e6e83dce2d1bce1]) successfully deleted
I20260928 12:26:08.289887 9248 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7b05960b8f406186f865b29e08d69a (DEFAULT_TABLE table=test-table [id=5874155f3d3045b5a5f969cf9a60b5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:08.289984 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b05960b8f406186f865b29e08d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:08.290227 9568 ts_tablet_manager.cc:1918] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:08.290321 9579 tablet_bootstrap.cc:492] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080: Bootstrap starting.
I20260928 12:26:08.290589 9579 tablet_bootstrap.cc:654] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:08.291024 9578 tablet_bootstrap.cc:492] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Bootstrap starting.
I20260928 12:26:08.290822 9577 tablet_bootstrap.cc:492] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Bootstrap starting.
I20260928 12:26:08.291103 9579 tablet_bootstrap.cc:492] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080: No bootstrap required, opened a new log
I20260928 12:26:08.291157 9579 ts_tablet_manager.cc:1397] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:08.291268 9579 raft_consensus.cc:348] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.291343 9578 tablet_bootstrap.cc:654] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:08.291350 9579 raft_consensus.cc:374] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:08.291374 9579 raft_consensus.cc:729] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83790ed29b1467e96954c0098e48080, State: Initialized, Role: FOLLOWER
I20260928 12:26:08.291404 9577 tablet_bootstrap.cc:654] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:08.291420 9579 consensus_queue.cc:260] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.291648 9579 ts_tablet_manager.cc:1428] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:08.291942 9568 log.cc:1196] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/219836a59f6844fba385858b517e70a6
I20260928 12:26:08.292088 9578 tablet_bootstrap.cc:492] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: No bootstrap required, opened a new log
I20260928 12:26:08.292153 9578 ts_tablet_manager.cc:1397] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:08.292179 9568 ts_tablet_manager.cc:1939] T 219836a59f6844fba385858b517e70a6 P 5c78767d81ba48cabd4e855ea0bc6956: Deleting consensus metadata
I20260928 12:26:08.292290 9578 raft_consensus.cc:348] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.292393 9578 raft_consensus.cc:374] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:08.292423 9578 raft_consensus.cc:729] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bc539e82144f559fae1e7fe82e7da9, State: Initialized, Role: FOLLOWER
I20260928 12:26:08.292464 9578 consensus_queue.cc:260] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.292470 8956 catalog_manager.cc:4841] TS 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601): tablet 219836a59f6844fba385858b517e70a6 (table test-table [id=423ba13aecda49ac9e6e83dce2d1bce1]) successfully deleted
I20260928 12:26:08.292568 9578 ts_tablet_manager.cc:1428] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:08.292762 9577 tablet_bootstrap.cc:492] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: No bootstrap required, opened a new log
I20260928 12:26:08.292810 9577 ts_tablet_manager.cc:1397] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:08.292917 9577 raft_consensus.cc:348] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.292979 9577 raft_consensus.cc:374] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:08.293010 9577 raft_consensus.cc:729] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c78767d81ba48cabd4e855ea0bc6956, State: Initialized, Role: FOLLOWER
I20260928 12:26:08.293056 9577 consensus_queue.cc:260] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.293143 9577 ts_tablet_manager.cc:1428] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:08.441581 9547 raft_consensus.cc:482] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:08.441697 9547 raft_consensus.cc:504] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.441908 9547 leader_election.cc:288] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:08.442095 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b05960b8f406186f865b29e08d69a" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956" is_pre_election: true
I20260928 12:26:08.442076 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b05960b8f406186f865b29e08d69a" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc539e82144f559fae1e7fe82e7da9" is_pre_election: true
I20260928 12:26:08.442185 9268 raft_consensus.cc:2454] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 0.
I20260928 12:26:08.442185 9408 raft_consensus.cc:2454] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 0.
I20260928 12:26:08.442342 9061 leader_election.cc:302] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bc539e82144f559fae1e7fe82e7da9, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:08.442476 9547 raft_consensus.cc:2781] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:08.442524 9547 raft_consensus.cc:482] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:08.442559 9547 raft_consensus.cc:3037] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:08.443238 9547 raft_consensus.cc:504] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.443382 9547 leader_election.cc:288] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 election: Requested vote from peers 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811), 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601)
I20260928 12:26:08.443526 9268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b05960b8f406186f865b29e08d69a" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91bc539e82144f559fae1e7fe82e7da9"
I20260928 12:26:08.443526 9408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b05960b8f406186f865b29e08d69a" candidate_uuid: "b83790ed29b1467e96954c0098e48080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c78767d81ba48cabd4e855ea0bc6956"
I20260928 12:26:08.443598 9408 raft_consensus.cc:3037] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:08.443598 9268 raft_consensus.cc:3037] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:08.444357 9408 raft_consensus.cc:2454] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 1.
I20260928 12:26:08.444357 9268 raft_consensus.cc:2454] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83790ed29b1467e96954c0098e48080 in term 1.
I20260928 12:26:08.444553 9060 leader_election.cc:302] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c78767d81ba48cabd4e855ea0bc6956, b83790ed29b1467e96954c0098e48080; no voters:
I20260928 12:26:08.444697 9547 raft_consensus.cc:2781] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:08.444751 9547 raft_consensus.cc:686] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [term 1 LEADER]: Becoming Leader. State: Replica: b83790ed29b1467e96954c0098e48080, State: Running, Role: LEADER
I20260928 12:26:08.444813 9547 consensus_queue.cc:237] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } }
I20260928 12:26:08.445389 8970 catalog_manager.cc:5495] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 reported cstate change: term changed from 0 to 1, leader changed from <none> to b83790ed29b1467e96954c0098e48080 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "b83790ed29b1467e96954c0098e48080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83790ed29b1467e96954c0098e48080" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 39877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:08.476884 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:08.474531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.7.19.65:39877)
0928 12:26:08.474690 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:08.474692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:08.474698 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:26:08.474723 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:08.474723 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:08.474732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:08.474802 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:08.474908 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.475393 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.476109 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.476360 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.476426 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:08.476570 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:08.476624 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:08.476704 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:08.476772 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:26:08.476795 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260928 12:26:08.480978 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:08.479056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:45369 (local address 127.7.19.65:39877)
0928 12:26:08.479090 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:08.479096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:08.479104 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:08.479154 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:08.479155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:08.479161 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:08.479205 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:08.479281 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.479708 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.480366 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.480535 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.480585 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:08.480755 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:08.480794 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:08.480849 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:08.480895 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:26:08.480913 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:26:08.481050 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:08.478927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.7.19.65:45369)
0928 12:26:08.479058 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:08.479073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:08.479114 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:08.479214 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:08.479214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:08.479215 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:08.479248 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:08.479249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.479764 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.479764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:08.480299 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:08.480300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:08.480599 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:08.480600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:08.480636 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:08.480639 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:08.480855 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:08.480857 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:08.480893 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 12:26:08.480937 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 12:26:08.482738 9268 raft_consensus.cc:1261] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Refusing update from remote peer b83790ed29b1467e96954c0098e48080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:08.482740 9408 raft_consensus.cc:1261] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Refusing update from remote peer b83790ed29b1467e96954c0098e48080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:08.482949 9544 consensus_queue.cc:1035] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91bc539e82144f559fae1e7fe82e7da9" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:08.483052 9547 consensus_queue.cc:1035] T 3d7b05960b8f406186f865b29e08d69a P b83790ed29b1467e96954c0098e48080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c78767d81ba48cabd4e855ea0bc6956" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:26:08.486734 9545 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 12:26:10.004530 8969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:10.004635 8969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:10.005554 8969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c198c71946284d97b5e5105ec55c6735: Sending DeleteTablet for 3 replicas of tablet 3d7b05960b8f406186f865b29e08d69a
I20260928 12:26:10.005846 9388 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7b05960b8f406186f865b29e08d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:10 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:60042
I20260928 12:26:10.005873 9107 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7b05960b8f406186f865b29e08d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:10 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:50698
I20260928 12:26:10.005915 9248 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7b05960b8f406186f865b29e08d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:10 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:42376
I20260928 12:26:10.006058 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 9043
I20260928 12:26:10.007797 9620 tablet_replica.cc:330] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: stopping tablet replica
I20260928 12:26:10.007863 9620 raft_consensus.cc:2229] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:10.007916 9620 raft_consensus.cc:2258] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:10.008040 9622 tablet_replica.cc:330] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: stopping tablet replica
I20260928 12:26:10.008114 9622 raft_consensus.cc:2229] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:10.008162 9622 raft_consensus.cc:2258] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:10.008378 9620 ts_tablet_manager.cc:1905] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:10.008419 9622 ts_tablet_manager.cc:1905] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:10.010500 9620 ts_tablet_manager.cc:1918] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:26:10.010500 9622 ts_tablet_manager.cc:1918] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 12:26:10.010576 9620 log.cc:1196] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/3d7b05960b8f406186f865b29e08d69a
I20260928 12:26:10.010576 9622 log.cc:1196] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/3d7b05960b8f406186f865b29e08d69a
I20260928 12:26:10.010852 9622 ts_tablet_manager.cc:1939] T 3d7b05960b8f406186f865b29e08d69a P 91bc539e82144f559fae1e7fe82e7da9: Deleting consensus metadata
I20260928 12:26:10.010852 9620 ts_tablet_manager.cc:1939] T 3d7b05960b8f406186f865b29e08d69a P 5c78767d81ba48cabd4e855ea0bc6956: Deleting consensus metadata
I20260928 12:26:10.011210 8957 catalog_manager.cc:4841] TS 91bc539e82144f559fae1e7fe82e7da9 (127.7.19.66:43811): tablet 3d7b05960b8f406186f865b29e08d69a (table test-table [id=5874155f3d3045b5a5f969cf9a60b5e3]) successfully deleted
I20260928 12:26:10.011210 8956 catalog_manager.cc:4841] TS 5c78767d81ba48cabd4e855ea0bc6956 (127.7.19.67:36601): tablet 3d7b05960b8f406186f865b29e08d69a (table test-table [id=5874155f3d3045b5a5f969cf9a60b5e3]) successfully deleted
W20260928 12:26:10.012956 8955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:39877 (error 108)
I20260928 12:26:10.013201 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 9183
I20260928 12:26:10.014118 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:10.013336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:39877 (local address 127.0.0.1:50728)
0928 12:26:10.014007 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:10.014063 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":61,"threads_started":1}
W20260928 12:26:10.014292 8955 catalog_manager.cc:4584] TS b83790ed29b1467e96954c0098e48080 (127.7.19.65:39877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d7b05960b8f406186f865b29e08d69a: Network error: Client connection negotiation failed: client connection to 127.7.19.65:39877: connect: Connection refused (error 111)
I20260928 12:26:10.019524 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 9323
I20260928 12:26:10.025132 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 8939
2026-09-28T12:26:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5935 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:10 dist-test-slave-cb7x krb5kdc[9627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:10 dist-test-slave-cb7x krb5kdc[9627](info): set up 2 sockets
Sep 28 12:26:10 dist-test-slave-cb7x krb5kdc[9627](info): commencing operation
krb5kdc: starting...
W20260928 12:26:11.072403 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:11.191640 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39093
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:11.278231 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:11.278504 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:11.278548 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:11.279873 9640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:11.279951 9640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:11.279979 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:11.280011 9640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:11.280047 9640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:11.281721 9640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:11.282032 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:11.282295 9640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:11.287531 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.287571 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.287851 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:11.287848 9640 server_base.cc:1034] running on GCE node
I20260928 12:26:11.288143 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:11.288445 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:11.289598 9640 hybrid_clock.cc:648] HybridClock initialized: now 1790598371289566 us; error 59 us; skew 500 ppm
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:11.292258 9640 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:11.292809 9640 webserver.cc:460] Webserver started at http://127.7.19.126:40203/ using document root <none> and password file <none>
I20260928 12:26:11.293043 9640 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:11.293088 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:11.293254 9640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:11.294391 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "52e4a1f51e294e0d87b975a4d481bb74"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "5174100b675da54df6588248a4687862"
server_key_iv: "923a43377f9765d3d9a13dd2c711327b"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.294770 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "52e4a1f51e294e0d87b975a4d481bb74"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "5174100b675da54df6588248a4687862"
server_key_iv: "923a43377f9765d3d9a13dd2c711327b"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.296137 9640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:11.297087 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.297384 9640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:11.297487 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "52e4a1f51e294e0d87b975a4d481bb74"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "5174100b675da54df6588248a4687862"
server_key_iv: "923a43377f9765d3d9a13dd2c711327b"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.297564 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:11.314093 9640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:11.315975 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:11.316128 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:11.320091 9640 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:39093
I20260928 12:26:11.320142 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:39093 every 8 connection(s)
I20260928 12:26:11.320523 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:11.321404 9708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:11.323681 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74: Bootstrap starting.
I20260928 12:26:11.324275 9708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:11.324573 9708 log.cc:824] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:11.325237 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74: No bootstrap required, opened a new log
I20260928 12:26:11.326648 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 9640
I20260928 12:26:11.326745 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:11.326936 7245 external_mini_cluster.cc:1428] Setting key 7b5e3a214d778f67dc72a8628e425248
I20260928 12:26:11.327719 9708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e4a1f51e294e0d87b975a4d481bb74" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39093 } }
I20260928 12:26:11.327881 9708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:11.327944 9708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e4a1f51e294e0d87b975a4d481bb74, State: Initialized, Role: FOLLOWER
I20260928 12:26:11.328075 9708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e4a1f51e294e0d87b975a4d481bb74" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39093 } }
I20260928 12:26:11.328176 9708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:11.328217 9708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:11.328258 9708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:11.329180 9708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e4a1f51e294e0d87b975a4d481bb74" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39093 } }
I20260928 12:26:11.329299 9708 leader_election.cc:302] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e4a1f51e294e0d87b975a4d481bb74; no voters:
I20260928 12:26:11.329452 9708 leader_election.cc:288] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:11.329485 9713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:11.329608 9713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [term 1 LEADER]: Becoming Leader. State: Replica: 52e4a1f51e294e0d87b975a4d481bb74, State: Running, Role: LEADER
I20260928 12:26:11.329715 9708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:11.329696 9713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e4a1f51e294e0d87b975a4d481bb74" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39093 } }
W20260928 12:26:11.331559 9727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 12:26:11.330972 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e4a1f51e294e0d87b975a4d481bb74. Latest consensus state: current_term: 1 leader_uuid: "52e4a1f51e294e0d87b975a4d481bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e4a1f51e294e0d87b975a4d481bb74" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39093 } } }
W20260928 12:26:11.331620 9727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:11.331651 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:11.331154 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e4a1f51e294e0d87b975a4d481bb74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e4a1f51e294e0d87b975a4d481bb74" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39093 } } }
I20260928 12:26:11.331794 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:11.331930 9729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:11.332937 9729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:11.334331 9729 catalog_manager.cc:1347] Generated new cluster ID: eadb83b35e644baeacd686ff3daccb8c
I20260928 12:26:11.334383 9729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:11.338150 9729 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:11.338572 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.327455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.7.19.126:39093)
0928 12:26:11.327615 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:11.327624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:11.327658 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:26:11.327753 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:11.327758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:11.327799 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:11.327946 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:11.328648 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.329729 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.330648 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.331273 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.331401 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.334391 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:11.334395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:11.334396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:11.334409 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:11.336360 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.336790 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.336793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.336794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.336911 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.337115 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.337116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.337116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.337431 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:11.337593 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:11.338041 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 12:26:11.338180 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260928 12:26:11.338675 9729 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:11.347242 9729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e4a1f51e294e0d87b975a4d481bb74: Generated new TSK 0
I20260928 12:26:11.347445 9729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:11.376322 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39093
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:11.458442 9734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:11.458676 9734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:11.458712 9734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:11.459867 9734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:11.459934 9734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:11.459980 9734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:11.461438 9734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:11.461699 9734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:11.461935 9734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:11.466876 9742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.466876 9739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.466922 9740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:11.466946 9734 server_base.cc:1034] running on GCE node
I20260928 12:26:11.467262 9734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:11.467465 9734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:11.468611 9734 hybrid_clock.cc:648] HybridClock initialized: now 1790598371468600 us; error 27 us; skew 500 ppm
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:11.471212 9734 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:11.471580 9734 webserver.cc:460] Webserver started at http://127.7.19.65:36717/ using document root <none> and password file <none>
I20260928 12:26:11.471760 9734 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:11.471807 9734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:11.471894 9734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:11.472962 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "c30297f423d54fa5a10189db4d622b03"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "88df4e185162621541c257ce9b79624e"
server_key_iv: "91bd0754f4abbb34edf945564e1fee38"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.473315 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "c30297f423d54fa5a10189db4d622b03"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "88df4e185162621541c257ce9b79624e"
server_key_iv: "91bd0754f4abbb34edf945564e1fee38"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.474646 9734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:11.475445 9749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.475652 9734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:11.475761 9734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "c30297f423d54fa5a10189db4d622b03"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "88df4e185162621541c257ce9b79624e"
server_key_iv: "91bd0754f4abbb34edf945564e1fee38"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.475855 9734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:11.483920 9734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:11.486094 9734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:11.486256 9734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:11.486493 9734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:11.486812 9734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:11.486866 9734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.486917 9734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:11.486945 9734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.492558 9734 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:42581
I20260928 12:26:11.492592 9862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:42581 every 8 connection(s)
I20260928 12:26:11.492973 9734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:11.501472 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 9734
I20260928 12:26:11.501569 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:11.501760 7245 external_mini_cluster.cc:1428] Setting key a2f564327b48483f6be87de4b1534864
I20260928 12:26:11.502751 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.493722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:57093 (local address 127.7.19.126:39093)
0928 12:26:11.493828 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:11.493831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:11.493990 (+ 159us) server_negotiation.cc:407] Connection header received
0928 12:26:11.494553 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:11.494554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:11.494570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:11.494633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:11.495586 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.496107 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.497172 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.497405 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.497486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.499578 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:11.499580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:11.499581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:11.499591 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:11.501173 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.501685 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.501686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.501686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.501750 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.502045 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.502046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.502046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.502172 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:11.502227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:11.502575 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 12:26:11.502659 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 12:26:11.502959 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.493689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39093 (local address 127.7.19.65:57093)
0928 12:26:11.493864 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:11.493907 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:11.494441 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:11.494715 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:11.494718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:11.495291 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:11.495482 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:11.495487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.496205 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.496206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:11.497081 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:11.497083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.497490 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.497491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:11.497614 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.498068 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:11.498076 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:11.499479 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:11.501278 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:11.501284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:11.501287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:11.501609 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:11.501825 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:11.501826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:11.501827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:11.501970 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:11.502237 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:11.502250 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:11.502413 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:26:11.502553 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 12:26:11.502715 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260928 12:26:11.503481 9863 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39093
I20260928 12:26:11.503566 9863 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:11.503762 9863 heartbeater.cc:511] Master 127.7.19.126:39093 requested a full tablet report, sending...
I20260928 12:26:11.504364 9672 ts_manager.cc:194] Registered new tserver with Master: c30297f423d54fa5a10189db4d622b03 (127.7.19.65:42581)
I20260928 12:26:11.505061 9672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:57093
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:11.527061 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39093
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:11.606977 9868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:11.607170 9868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:11.607193 9868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:11.608249 9868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:11.608283 9868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:11.608307 9868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:11.609639 9868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:11.609819 9868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:11.609974 9868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:11.614943 9876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.614979 9873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.614948 9874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:11.615216 9868 server_base.cc:1034] running on GCE node
I20260928 12:26:11.615437 9868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:11.615686 9868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:11.616870 9868 hybrid_clock.cc:648] HybridClock initialized: now 1790598371616841 us; error 40 us; skew 500 ppm
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:11.619531 9868 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:11.619930 9868 webserver.cc:460] Webserver started at http://127.7.19.66:40865/ using document root <none> and password file <none>
I20260928 12:26:11.620124 9868 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:11.620168 9868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:11.620263 9868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:11.621419 9868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "1e7b4a81bf2448eaacb5d355616b0c3b"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "27e88157d5024c3c3e2e0d482f335589"
server_key_iv: "b029497345221bdf0d929dffeefe4ba6"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.621752 9868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "1e7b4a81bf2448eaacb5d355616b0c3b"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "27e88157d5024c3c3e2e0d482f335589"
server_key_iv: "b029497345221bdf0d929dffeefe4ba6"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.623065 9868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:11.623754 9883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.623916 9868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:11.624001 9868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "1e7b4a81bf2448eaacb5d355616b0c3b"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "27e88157d5024c3c3e2e0d482f335589"
server_key_iv: "b029497345221bdf0d929dffeefe4ba6"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.624066 9868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:11.650581 9868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:11.652515 9868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:11.652714 9868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:11.652947 9868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:11.653250 9868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:11.653281 9868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.653337 9868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:11.653360 9868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.658820 9868 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:46657
I20260928 12:26:11.658906 9996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:46657 every 8 connection(s)
I20260928 12:26:11.659209 9868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:11.661871 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 9868
I20260928 12:26:11.661962 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:11.662134 7245 external_mini_cluster.cc:1428] Setting key 0dc2ab7dff2866161404276205197fa3
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:11.669699 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.659899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:52291 (local address 127.7.19.126:39093)
0928 12:26:11.659962 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:11.659964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:11.660166 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:26:11.660709 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:11.660710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:11.660723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:11.660787 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:11.661721 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.662468 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.663651 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.664011 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.664094 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.666807 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:11.666809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:11.666809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:11.666815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:11.668248 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.668748 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.668749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.668750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.668798 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.669090 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.669090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.669091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.669187 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:11.669237 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:11.669572 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 12:26:11.669637 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:11.669991 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.659884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39093 (local address 127.7.19.66:52291)
0928 12:26:11.660040 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:11.660087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:11.660584 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:11.660867 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:11.660876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:11.661418 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:11.661597 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:11.661601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.662560 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.662561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:11.663557 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:11.663560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.664112 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.664113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:11.664269 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.664846 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:11.664855 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:11.666714 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:11.668332 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:11.668339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:11.668341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:11.668642 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:11.668859 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:11.668860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:11.668860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:11.669011 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:11.669255 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:11.669262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:11.669424 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 12:26:11.669557 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 12:26:11.669738 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 12:26:11.670516 9997 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39093
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:26:11.670621 9997 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:11.670825 9997 heartbeater.cc:511] Master 127.7.19.126:39093 requested a full tablet report, sending...
I20260928 12:26:11.671172 9672 ts_manager.cc:194] Registered new tserver with Master: 1e7b4a81bf2448eaacb5d355616b0c3b (127.7.19.66:46657)
I20260928 12:26:11.671530 9672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:52291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:11.687747 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39093
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:11.766835 10002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:11.767056 10002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:11.767094 10002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:11.768234 10002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:11.768299 10002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:11.768342 10002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:11.769747 10002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:11.769999 10002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:11.770212 10002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:11.775060 10008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.775107 10010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:11.775087 10007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:11.775216 10002 server_base.cc:1034] running on GCE node
I20260928 12:26:11.775490 10002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:11.775732 10002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:11.776911 10002 hybrid_clock.cc:648] HybridClock initialized: now 1790598371776886 us; error 54 us; skew 500 ppm
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:11.779340 10002 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:11.779816 10002 webserver.cc:460] Webserver started at http://127.7.19.67:37423/ using document root <none> and password file <none>
I20260928 12:26:11.780066 10002 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:11.780114 10002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:11.780256 10002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:11.781344 10002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "94b1272f99d0420c813c584bcdeee07b"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "ed359050ab970fca07899ba7e8df7b53"
server_key_iv: "60b82e02321915ccdd58b9807a8161be"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.781702 10002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "94b1272f99d0420c813c584bcdeee07b"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "ed359050ab970fca07899ba7e8df7b53"
server_key_iv: "60b82e02321915ccdd58b9807a8161be"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.783012 10002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:11.783859 10017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.784073 10002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:11.784168 10002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "94b1272f99d0420c813c584bcdeee07b"
format_stamp: "Formatted at 2026-09-28 12:26:11 on dist-test-slave-cb7x"
server_key: "ed359050ab970fca07899ba7e8df7b53"
server_key_iv: "60b82e02321915ccdd58b9807a8161be"
server_key_version: "encryptionkey@0"
I20260928 12:26:11.784248 10002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:11.793659 10002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:11.795473 10002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:11.795629 10002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:11.795837 10002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:11.796129 10002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:11.796182 10002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.796221 10002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:11.796249 10002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:11.802321 10002 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:32929
I20260928 12:26:11.802404 10130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:32929 every 8 connection(s)
I20260928 12:26:11.802718 10002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598371, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:11.812062 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.803454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:39165 (local address 127.7.19.126:39093)
0928 12:26:11.803680 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:11.803683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:11.803848 (+ 165us) server_negotiation.cc:407] Connection header received
0928 12:26:11.804412 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:11.804414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:11.804426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:11.804477 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:11.805422 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.805922 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.806917 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.807124 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.807211 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.809354 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:11.809356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:11.809356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:11.809364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:11.810640 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.811107 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.811108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.811109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.811170 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:11.811426 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:11.811427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:11.811427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:11.811541 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:11.811615 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:11.811891 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 12:26:11.811970 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260928 12:26:11.812275 10133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:11.803399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39093 (local address 127.7.19.67:39165)
0928 12:26:11.803698 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:11.803742 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:11.804280 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:11.804544 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:11.804548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:11.805130 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:11.805315 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:11.805321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.806018 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.806019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:11.806811 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:11.806822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:11.807219 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:11.807220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:11.807354 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:11.807832 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:11.807841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:11.809252 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:11.810737 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:11.810743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:11.810745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:11.810984 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:11.811247 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:11.811248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:11.811248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:11.811353 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:11.811616 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:11.811622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:11.811757 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 12:26:11.811870 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 12:26:11.812026 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":83,"threads_started":1}
I20260928 12:26:11.812310 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10002
I20260928 12:26:11.812407 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:11.812618 7245 external_mini_cluster.cc:1428] Setting key c71fba7a81bd25e02da3b18dc2f55179
I20260928 12:26:11.812811 10131 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39093
I20260928 12:26:11.812908 10131 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:11.813095 10131 heartbeater.cc:511] Master 127.7.19.126:39093 requested a full tablet report, sending...
I20260928 12:26:11.813488 9672 ts_manager.cc:194] Registered new tserver with Master: 94b1272f99d0420c813c584bcdeee07b (127.7.19.67:32929)
I20260928 12:26:11.813876 9672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:39165
I20260928 12:26:11.824256 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:26:11.835405 9712 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:11.834519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.7.19.126:39093)
0928 12:26:11.834582 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:11.834584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:11.834697 (+ 113us) server_negotiation.cc:407] Connection header received
0928 12:26:11.834737 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:11.834737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:11.834746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:11.834793 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:11.835288 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50222: BlockingRecv error: recv got EOF from 127.0.0.1:50222 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:11.836211 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 9734
I20260928 12:26:11.840660 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 9868
I20260928 12:26:11.844743 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10002
I20260928 12:26:11.848941 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 9640
2026-09-28T12:26:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1822 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[10143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[10143](info): set up 2 sockets
Sep 28 12:26:11 dist-test-slave-cb7x krb5kdc[10143](info): commencing operation
krb5kdc: starting...
W20260928 12:26:12.880491 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:12 dist-test-slave-cb7x krb5kdc[10143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:13.000182 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37865
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:13.083360 10156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:13.083596 10156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:13.083635 10156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:13.084894 10156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:13.084964 10156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:13.084986 10156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:13.085005 10156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:13.085035 10156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:13.086553 10156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:13.086818 10156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:13.087035 10156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:13.091925 10164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:13.091933 10162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:13.091933 10161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:13.092020 10156 server_base.cc:1034] running on GCE node
I20260928 12:26:13.092373 10156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:13.092720 10156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:13.093904 10156 hybrid_clock.cc:648] HybridClock initialized: now 1790598373093885 us; error 50 us; skew 500 ppm
Sep 28 12:26:13 dist-test-slave-cb7x krb5kdc[10143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598373, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:13.096426 10156 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:13.096922 10156 webserver.cc:460] Webserver started at http://127.7.19.126:45581/ using document root <none> and password file <none>
I20260928 12:26:13.097105 10156 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:13.097143 10156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:13.097245 10156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:13.098251 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "3a7b355bab934e37aff675caa21f5477"
format_stamp: "Formatted at 2026-09-28 12:26:13 on dist-test-slave-cb7x"
server_key: "70ee6c02ce894dff897ab0ae42263d1c"
server_key_iv: "69c9711267334da102366745b9c18053"
server_key_version: "encryptionkey@0"
I20260928 12:26:13.098549 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "3a7b355bab934e37aff675caa21f5477"
format_stamp: "Formatted at 2026-09-28 12:26:13 on dist-test-slave-cb7x"
server_key: "70ee6c02ce894dff897ab0ae42263d1c"
server_key_iv: "69c9711267334da102366745b9c18053"
server_key_version: "encryptionkey@0"
I20260928 12:26:13.099838 10156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:13.100595 10171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:13.100827 10156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:13.100934 10156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "3a7b355bab934e37aff675caa21f5477"
format_stamp: "Formatted at 2026-09-28 12:26:13 on dist-test-slave-cb7x"
server_key: "70ee6c02ce894dff897ab0ae42263d1c"
server_key_iv: "69c9711267334da102366745b9c18053"
server_key_version: "encryptionkey@0"
I20260928 12:26:13.101023 10156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:13.127019 10156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:13.129313 10156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:13.129521 10156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:13.133401 10156 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:37865
I20260928 12:26:13.133458 10223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:37865 every 8 connection(s)
I20260928 12:26:13.133816 10156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:13.134717 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:13.134791 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10156
I20260928 12:26:13.134893 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:13.135072 7245 external_mini_cluster.cc:1428] Setting key 5ac44628e4a367d5a3509a84680c1736
I20260928 12:26:13.138304 10224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477: Bootstrap starting.
I20260928 12:26:13.138984 10224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:13.139276 10224 log.cc:824] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:13.139997 10224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477: No bootstrap required, opened a new log
Sep 28 12:26:13 dist-test-slave-cb7x krb5kdc[10143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:13.142232 10224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b355bab934e37aff675caa21f5477" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37865 } }
I20260928 12:26:13.142386 10224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:13.142426 10224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7b355bab934e37aff675caa21f5477, State: Initialized, Role: FOLLOWER
I20260928 12:26:13.142554 10224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b355bab934e37aff675caa21f5477" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37865 } }
I20260928 12:26:13.142625 10224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:13.142648 10224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:13.142714 10224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:13.143415 10224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b355bab934e37aff675caa21f5477" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37865 } }
I20260928 12:26:13.143529 10224 leader_election.cc:302] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a7b355bab934e37aff675caa21f5477; no voters:
I20260928 12:26:13.143684 10224 leader_election.cc:288] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:13.143790 10229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:13.143954 10229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7b355bab934e37aff675caa21f5477, State: Running, Role: LEADER
I20260928 12:26:13.143970 10224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:13.144181 10229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b355bab934e37aff675caa21f5477" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37865 } }
I20260928 12:26:13.144629 10231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a7b355bab934e37aff675caa21f5477. Latest consensus state: current_term: 1 leader_uuid: "3a7b355bab934e37aff675caa21f5477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b355bab934e37aff675caa21f5477" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37865 } } }
I20260928 12:26:13.144740 10231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:13.145249 10238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:13.145262 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a7b355bab934e37aff675caa21f5477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b355bab934e37aff675caa21f5477" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37865 } } }
I20260928 12:26:13.145637 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:13.145856 10238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:13.146605 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:13.135585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.7.19.126:37865)
0928 12:26:13.135788 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:13.135798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:13.135834 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:26:13.136520 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:13.136529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:13.136592 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:13.136755 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:13.137344 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:13.138726 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:13.139559 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:13.140101 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:13.140212 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:13.142899 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:13.142903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:13.142904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:13.142914 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:13.144808 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:13.145223 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:13.145237 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:13.145237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:13.145364 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:13.145595 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:13.145596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:13.145596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:13.145890 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:13.146076 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:13.146187 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:26:13.146442 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260928 12:26:13.147333 10238 catalog_manager.cc:1347] Generated new cluster ID: c03c1bfcdd094dd7b0965e68c8d982ea
I20260928 12:26:13.147382 10238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:13.147477 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:26:13.152884 10238 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:13.153637 10238 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:13.160580 10238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a7b355bab934e37aff675caa21f5477: Generated new TSK 0
I20260928 12:26:13.160799 10238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:13.165663 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:13.161664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.7.19.126:37865)
0928 12:26:13.161708 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:13.161710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:13.161812 (+ 102us) server_negotiation.cc:407] Connection header received
0928 12:26:13.161889 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:13.161890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:13.161898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:13.161943 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:13.162359 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:13.162844 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:13.164166 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:13.164384 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:13.164443 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:13.165116 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:13.165117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:13.165117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:13.165402 (+ 285us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:13.165417 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:13.165423 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:13.165472 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:13.165559 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:26:13.165581 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:13.168062 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10156
2026-09-28T12:26:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1316 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:13 dist-test-slave-cb7x krb5kdc[10257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:13 dist-test-slave-cb7x krb5kdc[10257](info): set up 2 sockets
Sep 28 12:26:13 dist-test-slave-cb7x krb5kdc[10257](info): commencing operation
krb5kdc: starting...
W20260928 12:26:14.208565 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:14.325459 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39301
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:14.404261 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:14.404472 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:14.404496 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:14.405586 10270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:14.405622 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:14.405637 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:14.405648 10270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:14.405661 10270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:14.407027 10270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:39301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:14.407212 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:14.407379 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:14.412272 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.412329 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.412276 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:14.412346 10270 server_base.cc:1034] running on GCE node
I20260928 12:26:14.412765 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:14.413091 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:14.414278 10270 hybrid_clock.cc:648] HybridClock initialized: now 1790598374414257 us; error 38 us; skew 500 ppm
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:14.416885 10270 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:14.417454 10270 webserver.cc:460] Webserver started at http://127.7.19.126:37095/ using document root <none> and password file <none>
I20260928 12:26:14.417711 10270 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:14.417755 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:14.417904 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:14.419000 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "69b90dc2479d4bc5a66a8bdd00438899"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "625646c2744fb4b2589c1d302a3d602e"
server_key_iv: "a3d7e6eedcce9e040a6ddb557839c028"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.419391 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "69b90dc2479d4bc5a66a8bdd00438899"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "625646c2744fb4b2589c1d302a3d602e"
server_key_iv: "a3d7e6eedcce9e040a6ddb557839c028"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.420753 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:14.421530 10285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.421725 10270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:14.421826 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "69b90dc2479d4bc5a66a8bdd00438899"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "625646c2744fb4b2589c1d302a3d602e"
server_key_iv: "a3d7e6eedcce9e040a6ddb557839c028"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.421917 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:14.441390 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:14.443236 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:14.443394 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:14.447397 10270 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:39301
I20260928 12:26:14.447436 10337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:39301 every 8 connection(s)
I20260928 12:26:14.447817 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:14.448834 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:14.449870 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10270
I20260928 12:26:14.449981 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:14.450168 7245 external_mini_cluster.cc:1428] Setting key 487c6ce85e659e9872b6371a00174a04
I20260928 12:26:14.451696 10338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899: Bootstrap starting.
I20260928 12:26:14.452292 10338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:14.452633 10338 log.cc:824] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:14.453437 10338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899: No bootstrap required, opened a new log
I20260928 12:26:14.455582 10338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b90dc2479d4bc5a66a8bdd00438899" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39301 } }
I20260928 12:26:14.455741 10338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:14.455780 10338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b90dc2479d4bc5a66a8bdd00438899, State: Initialized, Role: FOLLOWER
I20260928 12:26:14.455932 10338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b90dc2479d4bc5a66a8bdd00438899" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39301 } }
I20260928 12:26:14.456033 10338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:14.456074 10338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:14.456126 10338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:14.456647 10338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b90dc2479d4bc5a66a8bdd00438899" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39301 } }
I20260928 12:26:14.456782 10338 leader_election.cc:302] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69b90dc2479d4bc5a66a8bdd00438899; no voters:
I20260928 12:26:14.456977 10338 leader_election.cc:288] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:14.457062 10343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:14.457230 10343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [term 1 LEADER]: Becoming Leader. State: Replica: 69b90dc2479d4bc5a66a8bdd00438899, State: Running, Role: LEADER
I20260928 12:26:14.457238 10338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:14.457446 10343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b90dc2479d4bc5a66a8bdd00438899" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39301 } }
I20260928 12:26:14.458906 10345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69b90dc2479d4bc5a66a8bdd00438899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b90dc2479d4bc5a66a8bdd00438899" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39301 } } }
I20260928 12:26:14.458989 10345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:14.459219 10344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69b90dc2479d4bc5a66a8bdd00438899. Latest consensus state: current_term: 1 leader_uuid: "69b90dc2479d4bc5a66a8bdd00438899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b90dc2479d4bc5a66a8bdd00438899" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39301 } } }
W20260928 12:26:14.459257 10358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 12:26:14.459290 10344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899 [sys.catalog]: This master's current role is: LEADER
W20260928 12:26:14.459367 10358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:14.459486 10360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:14.459653 10360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:14.461100 10360 catalog_manager.cc:1347] Generated new cluster ID: eaadb27d3d3543c4bc49a15837fe4fd2
I20260928 12:26:14.461151 10360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:14.461905 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.450732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.7.19.126:39301)
0928 12:26:14.450923 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:14.450932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:14.450964 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:26:14.451471 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:14.451481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:14.451541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:14.451695 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:14.452401 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.453773 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.454530 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.455200 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.455326 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:14.458056 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:14.458060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:14.458061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:14.458072 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:14.460058 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:14.460470 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:14.460473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:14.460473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:14.460614 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:14.460856 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:14.460857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:14.460857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:14.461156 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:14.461311 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:14.461569 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 12:26:14.461717 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260928 12:26:14.478792 10360 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:14.479362 10360 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:14.492255 10360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69b90dc2479d4bc5a66a8bdd00438899: Generated new TSK 0
I20260928 12:26:14.492466 10360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:14.528053 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39301
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:14.614985 10364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:14.615185 10364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:14.615209 10364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:14.616371 10364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:14.616412 10364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:14.616438 10364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:14.617919 10364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:14.618156 10364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:14.618389 10364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:14.623296 10369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.623270 10372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.623342 10370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:14.623447 10364 server_base.cc:1034] running on GCE node
I20260928 12:26:14.623636 10364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:14.623867 10364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:14.625042 10364 hybrid_clock.cc:648] HybridClock initialized: now 1790598374625022 us; error 37 us; skew 500 ppm
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:14.627609 10364 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:14.628007 10364 webserver.cc:460] Webserver started at http://127.7.19.65:33399/ using document root <none> and password file <none>
I20260928 12:26:14.628223 10364 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:14.628286 10364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:14.628427 10364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:14.629467 10364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "067e0711163c4ce6a6446c8a5be7bac8"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "055c61eb12c597e1624069b642accfbf"
server_key_iv: "ba44664a86083f2cd663e95e1833141c"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.629822 10364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "067e0711163c4ce6a6446c8a5be7bac8"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "055c61eb12c597e1624069b642accfbf"
server_key_iv: "ba44664a86083f2cd663e95e1833141c"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.631071 10364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:26:14.631742 10379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.631910 10364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:14.632002 10364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "067e0711163c4ce6a6446c8a5be7bac8"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "055c61eb12c597e1624069b642accfbf"
server_key_iv: "ba44664a86083f2cd663e95e1833141c"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.632081 10364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:14.656225 10364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:14.658614 10364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:14.658766 10364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:14.659021 10364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:14.659330 10364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:14.659381 10364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.659421 10364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:14.659456 10364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.665063 10364 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:40111
I20260928 12:26:14.665143 10492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:40111 every 8 connection(s)
I20260928 12:26:14.665445 10364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:14.673302 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10364
I20260928 12:26:14.673405 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:14.673612 7245 external_mini_cluster.cc:1428] Setting key 2f764bc138efbdcb486a439c6886e595
I20260928 12:26:14.674361 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.666131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:47795 (local address 127.7.19.126:39301)
0928 12:26:14.666197 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:14.666199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:14.666410 (+ 211us) server_negotiation.cc:407] Connection header received
0928 12:26:14.666899 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:14.666900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:14.666910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:14.666968 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:14.667838 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.668336 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.669405 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.669613 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.669667 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:14.671617 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:14.671619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:14.671619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:14.671625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:14.673018 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:14.673364 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:14.673366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:14.673366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:14.673437 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:14.673657 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:14.673658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:14.673659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:14.673793 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:14.673857 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:14.674174 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 12:26:14.674270 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:26:14.674556 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.666097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39301 (local address 127.7.19.65:47795)
0928 12:26:14.666294 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:14.666338 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:14.666819 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:14.667022 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:14.667025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:14.667575 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:14.667763 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:14.667767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.668435 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.668437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:14.669330 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:14.669332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.669686 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.669687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:14.669789 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:14.670209 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:14.670217 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:14.671537 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:14.673086 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:14.673092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:14.673093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:14.673292 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:14.673498 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:14.673498 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:14.673498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:14.673592 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:14.673867 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:14.673873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:14.674045 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 12:26:14.674155 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:26:14.674318 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260928 12:26:14.675884 10493 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39301
I20260928 12:26:14.675988 10493 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:14.676173 10493 heartbeater.cc:511] Master 127.7.19.126:39301 requested a full tablet report, sending...
I20260928 12:26:14.676740 10302 ts_manager.cc:194] Registered new tserver with Master: 067e0711163c4ce6a6446c8a5be7bac8 (127.7.19.65:40111)
I20260928 12:26:14.677649 10302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:47795
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:14.700315 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39301
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:14.781658 10498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:14.781853 10498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:14.781875 10498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:14.782953 10498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:14.782984 10498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:14.783010 10498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:14.784454 10498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:14.784631 10498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:14.784844 10498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:14.789500 10504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:14.789608 10498 server_base.cc:1034] running on GCE node
W20260928 12:26:14.789544 10503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.789523 10506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:14.789884 10498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:14.790123 10498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:14.791280 10498 hybrid_clock.cc:648] HybridClock initialized: now 1790598374791269 us; error 42 us; skew 500 ppm
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:14.793776 10498 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:14.794231 10498 webserver.cc:460] Webserver started at http://127.7.19.66:42513/ using document root <none> and password file <none>
I20260928 12:26:14.794461 10498 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:14.794528 10498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:14.794656 10498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:14.795670 10498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "0eebdd1ee080489fb9f2eb64a66e37e1"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "b9da5b77d614b58f24c6168de6dd3ed6"
server_key_iv: "6de3b69fe9fabcac939d7be2a0bb5362"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.796022 10498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "0eebdd1ee080489fb9f2eb64a66e37e1"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "b9da5b77d614b58f24c6168de6dd3ed6"
server_key_iv: "6de3b69fe9fabcac939d7be2a0bb5362"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.797366 10498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:14.798067 10513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.798229 10498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:14.798321 10498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "0eebdd1ee080489fb9f2eb64a66e37e1"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "b9da5b77d614b58f24c6168de6dd3ed6"
server_key_iv: "6de3b69fe9fabcac939d7be2a0bb5362"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.798401 10498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:14.816159 10498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:14.818452 10498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:14.818606 10498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:14.818819 10498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:14.819110 10498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:14.819167 10498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.819208 10498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:14.819234 10498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.825232 10498 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:42905
I20260928 12:26:14.825287 10626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:42905 every 8 connection(s)
I20260928 12:26:14.825604 10498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:14.834434 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.826289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:59031 (local address 127.7.19.126:39301)
0928 12:26:14.826349 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:14.826351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:14.826555 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:26:14.827075 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:14.827076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:14.827089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:14.827141 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:14.828059 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.828577 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.829579 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.829778 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.829862 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:14.831816 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:14.831818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:14.831818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:14.831826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:14.833045 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:14.833439 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:14.833441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:14.833441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:14.833498 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:14.833782 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:14.833783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:14.833783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:14.833892 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:14.833980 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:14.834244 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 12:26:14.834319 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:26:14.834583 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.826278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39301 (local address 127.7.19.66:59031)
0928 12:26:14.826435 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:14.826473 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:14.826966 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:14.827210 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:14.827214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:14.827783 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:14.827963 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:14.827967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.828697 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.828698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:14.829485 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:14.829487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.829870 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.829871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:14.829993 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:14.830396 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:14.830404 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:14.831721 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:14.833135 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:14.833140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:14.833142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:14.833351 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:14.833605 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:14.833606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:14.833606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:14.833707 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:14.833974 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:14.833979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:14.834110 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 12:26:14.834212 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:26:14.834358 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260928 12:26:14.834770 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10498
I20260928 12:26:14.834877 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:14.835059 10627 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39301
I20260928 12:26:14.835085 7245 external_mini_cluster.cc:1428] Setting key 93f0715dfc3e9fa50eec3ca7ccf714fc
I20260928 12:26:14.835153 10627 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:14.835314 10627 heartbeater.cc:511] Master 127.7.19.126:39301 requested a full tablet report, sending...
I20260928 12:26:14.835717 10302 ts_manager.cc:194] Registered new tserver with Master: 0eebdd1ee080489fb9f2eb64a66e37e1 (127.7.19.66:42905)
I20260928 12:26:14.836086 10302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:59031
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:14.858963 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39301
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:14.937068 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:14.937296 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:14.937332 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:14.938473 10632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:14.938534 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:14.938575 10632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:14.939929 10632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:14.940189 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:14.940400 10632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:14.945302 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.945314 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:14.945322 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:14.945391 10632 server_base.cc:1034] running on GCE node
I20260928 12:26:14.945655 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:14.945847 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:14.946978 10632 hybrid_clock.cc:648] HybridClock initialized: now 1790598374946968 us; error 37 us; skew 500 ppm
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:14.949375 10632 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:14.949836 10632 webserver.cc:460] Webserver started at http://127.7.19.67:34617/ using document root <none> and password file <none>
I20260928 12:26:14.950062 10632 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:14.950107 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:14.950254 10632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:14.951313 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "06ad44bcaf524a7a88f72401b879c971"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "b768c103d8b312e7f4aed737ede17de5"
server_key_iv: "615ff54139e81cbd2e725ab9ae0a03fd"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.951673 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "06ad44bcaf524a7a88f72401b879c971"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "b768c103d8b312e7f4aed737ede17de5"
server_key_iv: "615ff54139e81cbd2e725ab9ae0a03fd"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.953030 10632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:14.953749 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.953938 10632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:14.954025 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "06ad44bcaf524a7a88f72401b879c971"
format_stamp: "Formatted at 2026-09-28 12:26:14 on dist-test-slave-cb7x"
server_key: "b768c103d8b312e7f4aed737ede17de5"
server_key_iv: "615ff54139e81cbd2e725ab9ae0a03fd"
server_key_version: "encryptionkey@0"
I20260928 12:26:14.954095 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:14.985335 10632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:14.987248 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:14.987407 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:14.987648 10632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:14.987960 10632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:14.988014 10632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.988056 10632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:14.988083 10632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:14.993885 10632 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:34607
I20260928 12:26:14.993925 10760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:34607 every 8 connection(s)
I20260928 12:26:14.994256 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:14 dist-test-slave-cb7x krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598374, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:15.003216 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.995000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:39351 (local address 127.7.19.126:39301)
0928 12:26:14.995063 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:14.995065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:14.995252 (+ 187us) server_negotiation.cc:407] Connection header received
0928 12:26:14.995753 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:14.995754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:14.995765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:14.995841 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:14.996764 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.997284 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.998218 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.998394 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.998764 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:15.000457 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:15.000458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:15.000458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:15.000465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:15.001907 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:15.002411 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:15.002412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:15.002412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:15.002461 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:15.002662 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:15.002662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:15.002662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:15.002757 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:15.002805 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:15.003022 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 12:26:15.003077 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:15.003270 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10632
I20260928 12:26:15.003289 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:14.994992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39301 (local address 127.7.19.67:39351)
0928 12:26:14.995132 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:14.995172 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:14.995641 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:14.995893 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:14.995896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:14.996446 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:14.996633 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:14.996637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.997363 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.997363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:14.998150 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:14.998151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:14.998469 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:14.998469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:14.998566 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:14.999030 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:14.999038 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:15.000365 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:15.001984 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:15.001989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:15.001991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:15.002350 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:15.002517 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:15.002517 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:15.002518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:15.002611 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:15.002807 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:15.002811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:15.002918 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 12:26:15.003000 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:15.003112 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 12:26:15.003347 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:15.003522 7245 external_mini_cluster.cc:1428] Setting key 9d42eb29f29938cdde84fd1dc7cb57cf
I20260928 12:26:15.003775 10761 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39301
I20260928 12:26:15.003896 10761 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:15.004125 10761 heartbeater.cc:511] Master 127.7.19.126:39301 requested a full tablet report, sending...
I20260928 12:26:15.004504 10302 ts_manager.cc:194] Registered new tserver with Master: 06ad44bcaf524a7a88f72401b879c971 (127.7.19.67:34607)
I20260928 12:26:15.004932 10302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:39351
I20260928 12:26:15.015295 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:26:15 dist-test-slave-cb7x krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598375, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 12:26:15 dist-test-slave-cb7x krb5kdc[10257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598375, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:15.037118 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:15.030292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.7.19.126:39301)
0928 12:26:15.030381 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:15.030383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:15.030446 (+ 63us) server_negotiation.cc:407] Connection header received
0928 12:26:15.030494 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:15.030495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:15.030504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:15.030558 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:15.031081 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:15.031558 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:15.032252 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:15.032428 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:15.032477 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:15.034658 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:15.034659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:15.034659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:15.034666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:15.036109 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:15.036391 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:15.036392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:15.036392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:15.036438 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:15.036579 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:15.036579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:15.036580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:15.036764 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:15.036840 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:15.036969 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 12:26:15.037030 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 12:26:15.037439 10302 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47582
I20260928 12:26:15.037992 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10364
I20260928 12:26:15.042353 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10498
I20260928 12:26:15.046705 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10632
I20260928 12:26:15.050844 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10270
2026-09-28T12:26:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1885 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:15 dist-test-slave-cb7x krb5kdc[10773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:15 dist-test-slave-cb7x krb5kdc[10773](info): set up 2 sockets
Sep 28 12:26:15 dist-test-slave-cb7x krb5kdc[10773](info): commencing operation
krb5kdc: starting...
W20260928 12:26:16.076514 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:16.199851 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37733
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:16.280426 10786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:16.280628 10786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:16.280653 10786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:16.281847 10786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:16.281890 10786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:16.281905 10786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:16.281917 10786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:16.281929 10786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:16.283449 10786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:16.283686 10786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:16.283937 10786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:16.288708 10792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.288790 10786 server_base.cc:1034] running on GCE node
W20260928 12:26:16.288728 10794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:16.288739 10791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.289110 10786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:16.289383 10786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:16.290544 10786 hybrid_clock.cc:648] HybridClock initialized: now 1790598376290540 us; error 46 us; skew 500 ppm
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:16.292917 10786 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:16.293334 10786 webserver.cc:460] Webserver started at http://127.7.19.126:42497/ using document root <none> and password file <none>
I20260928 12:26:16.293565 10786 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:16.293633 10786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:16.293782 10786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:16.294819 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "5f228d1e080ef70e0d023c6002f3197e"
server_key_iv: "c9bfe045dafa1774c854920d63e47de5"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.295347 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "5f228d1e080ef70e0d023c6002f3197e"
server_key_iv: "c9bfe045dafa1774c854920d63e47de5"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.296763 10786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:16.297536 10801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.297708 10786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:16.297812 10786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "5f228d1e080ef70e0d023c6002f3197e"
server_key_iv: "c9bfe045dafa1774c854920d63e47de5"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.297894 10786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:16.306061 10786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:16.307981 10786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:16.308137 10786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:16.312075 10786 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:37733
I20260928 12:26:16.312105 10853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:37733 every 8 connection(s)
I20260928 12:26:16.312574 10786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:16.313278 10854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:16.314301 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10786
I20260928 12:26:16.314390 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:16.314613 7245 external_mini_cluster.cc:1428] Setting key 7508a7342224dd242728164a28d93354
I20260928 12:26:16.316004 10854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61: Bootstrap starting.
I20260928 12:26:16.316609 10854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:16.316936 10854 log.cc:824] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:16.317595 10854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61: No bootstrap required, opened a new log
I20260928 12:26:16.319314 10854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37733 } }
I20260928 12:26:16.319468 10854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:16.319515 10854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aafb6e97dbc4aaa8a5f72b5463f3e61, State: Initialized, Role: FOLLOWER
I20260928 12:26:16.319669 10854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37733 } }
I20260928 12:26:16.319761 10854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:16.319806 10854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:16.319854 10854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:16.320363 10854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37733 } }
I20260928 12:26:16.320480 10854 leader_election.cc:302] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aafb6e97dbc4aaa8a5f72b5463f3e61; no voters:
I20260928 12:26:16.320650 10854 leader_election.cc:288] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:16.320727 10859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:16.320860 10859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [term 1 LEADER]: Becoming Leader. State: Replica: 4aafb6e97dbc4aaa8a5f72b5463f3e61, State: Running, Role: LEADER
I20260928 12:26:16.320935 10854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:16.320986 10859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37733 } }
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:16.321432 10860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aafb6e97dbc4aaa8a5f72b5463f3e61. Latest consensus state: current_term: 1 leader_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37733 } } }
I20260928 12:26:16.321471 10861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aafb6e97dbc4aaa8a5f72b5463f3e61" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37733 } } }
I20260928 12:26:16.321713 10861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:16.321604 10860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:16.321969 10868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:16.322476 10868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:16.324076 10868 catalog_manager.cc:1347] Generated new cluster ID: 00f42e429f0a47f393c0b492e408edea
I20260928 12:26:16.324128 10868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:16.325796 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.315121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.7.19.126:37733)
0928 12:26:16.315299 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:16.315309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:16.315340 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:26:16.315933 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:16.315941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:16.316001 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:16.316153 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:16.316808 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.317801 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.318537 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.318949 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.319066 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.321800 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:16.321804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:16.321805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:16.321818 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:16.324027 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.324393 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.324396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.324396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.324493 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.324652 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.324652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.324652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.324952 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:16.325074 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:16.325482 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 12:26:16.325610 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 12:26:16.354084 10868 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:16.354857 10868 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:16.366940 10868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aafb6e97dbc4aaa8a5f72b5463f3e61: Generated new TSK 0
I20260928 12:26:16.367164 10868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:16.399961 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:37733
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:16.481885 10880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:16.482085 10880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:16.482107 10880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:16.483274 10880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:16.483322 10880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:16.483349 10880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:16.485001 10880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:16.485251 10880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:16.485471 10880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:16.490553 10885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:16.490562 10886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.490739 10880 server_base.cc:1034] running on GCE node
W20260928 12:26:16.490568 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.491048 10880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:16.491283 10880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:16.492444 10880 hybrid_clock.cc:648] HybridClock initialized: now 1790598376492442 us; error 50 us; skew 500 ppm
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:16.495114 10880 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:16.495610 10880 webserver.cc:460] Webserver started at http://127.7.19.65:42039/ using document root <none> and password file <none>
I20260928 12:26:16.495841 10880 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:16.495913 10880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:16.496066 10880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:16.497241 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "08d4411a52d647a1948009982ce7c71d"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "e5ce4c3e635ce9824f740dc88dfe54ca"
server_key_iv: "e3f51b87a3f4d7a849269bd68464121f"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.497622 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "08d4411a52d647a1948009982ce7c71d"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "e5ce4c3e635ce9824f740dc88dfe54ca"
server_key_iv: "e3f51b87a3f4d7a849269bd68464121f"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.498987 10880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:16.499776 10895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.499979 10880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:16.500084 10880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "08d4411a52d647a1948009982ce7c71d"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "e5ce4c3e635ce9824f740dc88dfe54ca"
server_key_iv: "e3f51b87a3f4d7a849269bd68464121f"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.500166 10880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:16.525146 10880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:16.527249 10880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:16.527426 10880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:16.527667 10880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:16.528008 10880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:16.528064 10880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.528105 10880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:16.528132 10880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.533813 10880 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:38675
I20260928 12:26:16.533856 11008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:38675 every 8 connection(s)
I20260928 12:26:16.534166 10880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:16.534997 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 10880
I20260928 12:26:16.535101 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:16.535292 7245 external_mini_cluster.cc:1428] Setting key cfe466144976c3a8655e27e2a7d47ee0
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
I20260928 12:26:16.544201 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.534905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:41055 (local address 127.7.19.126:37733)
0928 12:26:16.534978 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:16.534981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:16.535178 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:26:16.536125 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:16.536126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:16.536138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:16.536185 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:16.536879 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.537390 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.538742 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.538948 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.539008 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.541436 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:16.541437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:16.541437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:16.541445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:16.542826 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.543246 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.543247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.543248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.543326 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.543570 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.543571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.543572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.543682 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:16.543751 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:16.544032 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 12:26:16.544114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:16.544471 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.534872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37733 (local address 127.7.19.65:41055)
0928 12:26:16.535031 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:16.535074 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:16.536015 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:16.536257 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:16.536269 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:16.536567 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:16.536775 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:16.536780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.537487 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.537489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:16.538658 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:16.538660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.539037 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.539038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:16.539192 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.539697 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:16.539708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:16.541344 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:16.542927 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:16.542934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:16.542936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:16.543155 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:16.543379 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:16.543380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:16.543381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:16.543496 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:16.543757 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:16.543762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:16.543895 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 12:26:16.543997 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:26:16.544144 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 12:26:16.545038 11009 heartbeater.cc:347] Connected to a master server at 127.7.19.126:37733
I20260928 12:26:16.545136 11009 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:16.545324 11009 heartbeater.cc:511] Master 127.7.19.126:37733 requested a full tablet report, sending...
I20260928 12:26:16.545907 10818 ts_manager.cc:194] Registered new tserver with Master: 08d4411a52d647a1948009982ce7c71d (127.7.19.65:38675)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
I20260928 12:26:16.546603 10818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:41055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:16.561044 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:37733
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:16.639914 11014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:16.640110 11014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:16.640132 11014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:16.641301 11014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:16.641341 11014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:16.641392 11014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:16.642858 11014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:16.643081 11014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:16.643306 11014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:16.648283 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:16.648341 11019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.648430 11014 server_base.cc:1034] running on GCE node
W20260928 12:26:16.648324 11020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.648774 11014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:16.649012 11014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:16.650151 11014 hybrid_clock.cc:648] HybridClock initialized: now 1790598376650141 us; error 26 us; skew 500 ppm
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:16.652745 11014 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:16.653195 11014 webserver.cc:460] Webserver started at http://127.7.19.66:46133/ using document root <none> and password file <none>
I20260928 12:26:16.653424 11014 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:16.653469 11014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:16.653616 11014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:16.654901 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "148da41c122e420298ea0076b73979aa"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "d16ea91036da6a82333ea10bc8da1abc"
server_key_iv: "4100412165d109af9a998f2bb415265e"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.655313 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "148da41c122e420298ea0076b73979aa"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "d16ea91036da6a82333ea10bc8da1abc"
server_key_iv: "4100412165d109af9a998f2bb415265e"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.657382 11014 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 12:26:16.658176 11029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.658349 11014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:16.658442 11014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "148da41c122e420298ea0076b73979aa"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "d16ea91036da6a82333ea10bc8da1abc"
server_key_iv: "4100412165d109af9a998f2bb415265e"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.658529 11014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:16.701296 11014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:16.703975 11014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:16.704147 11014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:16.704393 11014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:16.704763 11014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:16.704818 11014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.704857 11014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:16.704885 11014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.710459 11014 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:40555
I20260928 12:26:16.710500 11142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:40555 every 8 connection(s)
I20260928 12:26:16.710819 11014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:16.715698 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11014
I20260928 12:26:16.715822 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:16.716027 7245 external_mini_cluster.cc:1428] Setting key fb44833a1cf040a819148b21e2f03096
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:16.721851 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.711530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:55145 (local address 127.7.19.126:37733)
0928 12:26:16.711594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:16.711596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:16.711795 (+ 199us) server_negotiation.cc:407] Connection header received
0928 12:26:16.712783 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:16.712785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:16.712798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:16.712848 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:16.713467 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.714017 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.715076 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.715276 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.715368 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.718316 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:16.718318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:16.718318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:16.718327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:16.720035 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.720594 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.720595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.720596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.720703 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.721054 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.721054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.721055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.721190 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:16.721234 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:16.721603 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 12:26:16.721716 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:16.721978 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.711502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37733 (local address 127.7.19.66:55145)
0928 12:26:16.711670 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:16.711714 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:16.712644 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:16.712914 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:16.712918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:16.713170 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:16.713355 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:16.713359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.714112 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.714113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:16.714979 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:16.714981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.715373 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.715374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:16.715524 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.716183 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:16.716194 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:16.718197 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:16.720127 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:16.720134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:16.720136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:16.720482 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:16.720794 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:16.720795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:16.720795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:16.720962 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:16.721281 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:16.721286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:16.721456 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 12:26:16.721568 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:26:16.721734 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260928 12:26:16.723549 11143 heartbeater.cc:347] Connected to a master server at 127.7.19.126:37733
I20260928 12:26:16.723711 11143 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:16.723899 11143 heartbeater.cc:511] Master 127.7.19.126:37733 requested a full tablet report, sending...
I20260928 12:26:16.724515 10818 ts_manager.cc:194] Registered new tserver with Master: 148da41c122e420298ea0076b73979aa (127.7.19.66:40555)
I20260928 12:26:16.725093 10818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:55145
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:16.743933 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:37733
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:16.823904 11148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:16.824102 11148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:16.824123 11148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:16.825328 11148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:16.825369 11148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:16.825398 11148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:16.826956 11148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:37733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:16.827183 11148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:16.827399 11148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:16.832152 11153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:16.832167 11156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:16.832166 11154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:16.832324 11148 server_base.cc:1034] running on GCE node
I20260928 12:26:16.832623 11148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:16.832916 11148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:16.834117 11148 hybrid_clock.cc:648] HybridClock initialized: now 1790598376834103 us; error 25 us; skew 500 ppm
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:16.836711 11148 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:16.837157 11148 webserver.cc:460] Webserver started at http://127.7.19.67:34383/ using document root <none> and password file <none>
I20260928 12:26:16.837404 11148 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:16.837477 11148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:16.837637 11148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:16.838801 11148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "7c9a018b5b32497dbf9019a20d9797e4"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "5c5e94455e22acbd19c88c7db992b480"
server_key_iv: "48aea8f4996b67f165b368c97e309acd"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.839181 11148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "7c9a018b5b32497dbf9019a20d9797e4"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "5c5e94455e22acbd19c88c7db992b480"
server_key_iv: "48aea8f4996b67f165b368c97e309acd"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.840768 11148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:16.841600 11163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.841763 11148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:16.841859 11148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "7c9a018b5b32497dbf9019a20d9797e4"
format_stamp: "Formatted at 2026-09-28 12:26:16 on dist-test-slave-cb7x"
server_key: "5c5e94455e22acbd19c88c7db992b480"
server_key_iv: "48aea8f4996b67f165b368c97e309acd"
server_key_version: "encryptionkey@0"
I20260928 12:26:16.841940 11148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:16.850912 11148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:16.852948 11148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:16.853104 11148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:16.853327 11148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:16.853619 11148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:16.853672 11148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.853706 11148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:16.853734 11148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:16.859539 11148 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:39549
I20260928 12:26:16.859576 11276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:39549 every 8 connection(s)
I20260928 12:26:16.859895 11148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:16.868575 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11148
I20260928 12:26:16.868706 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:16.868930 7245 external_mini_cluster.cc:1428] Setting key 7674be6f7408869733e2a65793b89eaa
I20260928 12:26:16.869405 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.860654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:43209 (local address 127.7.19.126:37733)
0928 12:26:16.860755 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:16.860757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:16.860892 (+ 135us) server_negotiation.cc:407] Connection header received
0928 12:26:16.861881 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:16.861882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:16.861893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:16.861959 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:16.862600 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.863096 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.864023 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.864205 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.864260 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.866382 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:16.866383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:16.866384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:16.866390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:16.867891 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.868255 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.868255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.868256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.868306 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.868663 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.868663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.868664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.868829 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:16.868883 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:16.869235 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 12:26:16.869317 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 12:26:16.869601 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.860598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37733 (local address 127.7.19.67:43209)
0928 12:26:16.860769 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:16.860808 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:16.861742 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:16.862029 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:16.862032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:16.862320 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:16.862523 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:16.862528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.863168 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.863169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:16.863944 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:16.863946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.864274 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.864275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:16.864365 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.864969 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:16.864977 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:16.866295 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:16.867961 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:16.867967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:16.867968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:16.868184 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:16.868362 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:16.868363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:16.868414 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:16.868566 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:16.868891 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:16.868898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:16.869066 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 12:26:16.869193 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 12:26:16.869348 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
I20260928 12:26:16.870132 11277 heartbeater.cc:347] Connected to a master server at 127.7.19.126:37733
I20260928 12:26:16.870249 11277 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:16.870415 11277 heartbeater.cc:511] Master 127.7.19.126:37733 requested a full tablet report, sending...
I20260928 12:26:16.870824 10818 ts_manager.cc:194] Registered new tserver with Master: 7c9a018b5b32497dbf9019a20d9797e4 (127.7.19.67:39549)
I20260928 12:26:16.871204 10818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:43209
I20260928 12:26:16.880640 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:16.903170 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.895365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.7.19.65:38675)
0928 12:26:16.895552 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:16.895557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:16.895567 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:16.895593 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:16.895595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:16.895608 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:16.895672 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:16.896257 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.897018 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.897808 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.898113 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.898165 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.900396 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:16.900424 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:16.900424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:16.900442 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:16.902070 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.902469 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.902472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.902473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.902526 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.902715 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.902715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.902715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.902830 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:16.902927 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:16.902998 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 12:26:16.903057 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[10773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:16.910670 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:16.904485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.7.19.126:37733)
0928 12:26:16.904525 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:16.904527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:16.904572 (+ 45us) server_negotiation.cc:407] Connection header received
0928 12:26:16.904634 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:16.904635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:16.904643 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:16.904745 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:16.905187 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.905661 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.906286 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:16.906467 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:16.906523 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:16.908567 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:16.908568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:16.908568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:16.908575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:16.909763 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.910054 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.910055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.910055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.910103 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:16.910257 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:16.910257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:16.910258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:16.910387 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:16.910474 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:16.910551 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 12:26:16.910607 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:26:16.910904 10818 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50882
I20260928 12:26:16.911209 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10880
I20260928 12:26:16.916075 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11014
I20260928 12:26:16.920213 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11148
I20260928 12:26:16.924325 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 10786
2026-09-28T12:26:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1873 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[11289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[11289](info): set up 2 sockets
Sep 28 12:26:16 dist-test-slave-cb7x krb5kdc[11289](info): commencing operation
krb5kdc: starting...
W20260928 12:26:17.976729 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:18.096354 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46053
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:18.176846 11302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:18.177044 11302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:18.177066 11302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:18.178170 11302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:18.178208 11302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:18.178222 11302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:18.178241 11302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:18.178253 11302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:18.179653 11302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:18.179841 11302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:18.180006 11302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:18.184648 11308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.184657 11307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.184648 11310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:18.184912 11302 server_base.cc:1034] running on GCE node
I20260928 12:26:18.185210 11302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:18.185490 11302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:18.186664 11302 hybrid_clock.cc:648] HybridClock initialized: now 1790598378186704 us; error 100 us; skew 500 ppm
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:18.189160 11302 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:18.189527 11302 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:18.189577 11302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:18.189667 11302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:18.190672 11302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "878e812714ad46ba9f241eb7d0c0e212"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "f7a136a6a355da333d5f3a114cb906cf"
server_key_iv: "3b9b042812f42c0e8da04a9d9166471d"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.190985 11302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "878e812714ad46ba9f241eb7d0c0e212"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "f7a136a6a355da333d5f3a114cb906cf"
server_key_iv: "3b9b042812f42c0e8da04a9d9166471d"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.192222 11302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:18.192947 11316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.193172 11302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:18.193264 11302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "878e812714ad46ba9f241eb7d0c0e212"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "f7a136a6a355da333d5f3a114cb906cf"
server_key_iv: "3b9b042812f42c0e8da04a9d9166471d"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.193317 11302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:18.210985 11302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:18.213027 11302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:18.213140 11302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:18.217034 11302 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:46053
I20260928 12:26:18.217067 11368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:46053 every 8 connection(s)
I20260928 12:26:18.217375 11302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:18.218298 11369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:18.220727 11369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212: Bootstrap starting.
I20260928 12:26:18.221212 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11302
I20260928 12:26:18.221318 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:18.221510 7245 external_mini_cluster.cc:1428] Setting key dd8b1c8c897ff0191775103b66932ce5
I20260928 12:26:18.221534 11369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:18.221871 11369 log.cc:824] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:18.222599 11369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212: No bootstrap required, opened a new log
I20260928 12:26:18.224491 11369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878e812714ad46ba9f241eb7d0c0e212" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46053 } }
I20260928 12:26:18.224654 11369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:18.224725 11369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878e812714ad46ba9f241eb7d0c0e212, State: Initialized, Role: FOLLOWER
I20260928 12:26:18.224864 11369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878e812714ad46ba9f241eb7d0c0e212" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46053 } }
I20260928 12:26:18.224967 11369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:18.225011 11369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:18.225059 11369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:18.225641 11369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878e812714ad46ba9f241eb7d0c0e212" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46053 } }
I20260928 12:26:18.225764 11369 leader_election.cc:302] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 878e812714ad46ba9f241eb7d0c0e212; no voters:
I20260928 12:26:18.225953 11369 leader_election.cc:288] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:18.225992 11374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:18.226126 11374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [term 1 LEADER]: Becoming Leader. State: Replica: 878e812714ad46ba9f241eb7d0c0e212, State: Running, Role: LEADER
I20260928 12:26:18.226212 11374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878e812714ad46ba9f241eb7d0c0e212" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46053 } }
I20260928 12:26:18.226231 11369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:18.226887 11376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 878e812714ad46ba9f241eb7d0c0e212. Latest consensus state: current_term: 1 leader_uuid: "878e812714ad46ba9f241eb7d0c0e212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878e812714ad46ba9f241eb7d0c0e212" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46053 } } }
I20260928 12:26:18.226966 11376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:18.226985 11375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "878e812714ad46ba9f241eb7d0c0e212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878e812714ad46ba9f241eb7d0c0e212" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46053 } } }
I20260928 12:26:18.227031 11375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:18.227241 11384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:18.227735 11384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:18.229310 11384 catalog_manager.cc:1347] Generated new cluster ID: c5fbdca13e7740abbdda5991ce6168ca
I20260928 12:26:18.229416 11384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:18.234370 11384 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:18.235078 11384 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:18.239782 11384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212: Generated new TSK 0
I20260928 12:26:18.239955 11384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:18.241110 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.222026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.7.19.126:46053)
0928 12:26:18.222184 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.222194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.222225 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:26:18.222685 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.222694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.222770 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.222911 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:18.223517 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.224415 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.225404 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.225924 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.226039 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.237291 (+ 11252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:18.237296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:18.237296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:18.237309 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:18.239267 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.239704 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.239708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.239708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.239828 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.240049 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.240050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.240050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.240317 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:18.240463 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.240709 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.240873 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:18.276651 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46053
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:18.377611 11395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:18.377847 11395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:18.377885 11395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:18.379022 11395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:18.379083 11395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:18.379128 11395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:18.380615 11395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:18.380872 11395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:18.381127 11395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:18.386935 11401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.386971 11400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.386957 11403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:18.387117 11395 server_base.cc:1034] running on GCE node
I20260928 12:26:18.387312 11395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:18.387599 11395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:18.388733 11395 hybrid_clock.cc:648] HybridClock initialized: now 1790598378388728 us; error 41 us; skew 500 ppm
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:18.391534 11395 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:18.391917 11395 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:18.391970 11395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:18.392063 11395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:18.393206 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "47f9a718ed0841dea18914e7a3322b05"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "e45d3a82a4c06a66b02946b8bf11e884"
server_key_iv: "defc8e563571e2cee9b2bd2b30bc4bba"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.393559 11395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "47f9a718ed0841dea18914e7a3322b05"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "e45d3a82a4c06a66b02946b8bf11e884"
server_key_iv: "defc8e563571e2cee9b2bd2b30bc4bba"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.395054 11395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:18.395814 11409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.395992 11395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:18.396080 11395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "47f9a718ed0841dea18914e7a3322b05"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "e45d3a82a4c06a66b02946b8bf11e884"
server_key_iv: "defc8e563571e2cee9b2bd2b30bc4bba"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.396148 11395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:18.409546 11395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:18.411834 11395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:18.411985 11395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:18.412209 11395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:18.412520 11395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:18.412555 11395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.412577 11395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:18.412590 11395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.418483 11395 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:35035
I20260928 12:26:18.418526 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:35035 every 8 connection(s)
I20260928 12:26:18.418855 11395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:18.422386 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11395
I20260928 12:26:18.422487 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:18.422674 7245 external_mini_cluster.cc:1428] Setting key ce7710a88eea404c9a036c92953bc2ae
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:18.428870 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.419583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:41621 (local address 127.7.19.126:46053)
0928 12:26:18.419663 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.419666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.419759 (+ 93us) server_negotiation.cc:407] Connection header received
0928 12:26:18.420245 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.420247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.420258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.420315 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:18.421273 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.421790 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.423323 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.423547 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.423632 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.425815 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:18.425816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:18.425817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:18.425824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:18.427420 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.427815 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.427816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.427817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.427880 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.428212 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.428213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.428214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.428337 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:18.428413 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.428698 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.428781 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:26:18.429071 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.419499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46053 (local address 127.7.19.65:41621)
0928 12:26:18.419656 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.419693 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.420151 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.420387 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.420390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.420634 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:18.421190 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.421195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.421904 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.421905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.423238 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.423242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.423637 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.423638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.423765 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.424295 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:18.424303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:18.425710 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:18.427497 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:18.427504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:18.427506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:18.427729 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:18.427979 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:18.427980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:18.427981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:18.428141 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:18.428405 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:18.428411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:18.428549 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.428655 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.428824 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260928 12:26:18.429593 11523 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46053
I20260928 12:26:18.429672 11523 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:18.429847 11523 heartbeater.cc:511] Master 127.7.19.126:46053 requested a full tablet report, sending...
I20260928 12:26:18.430403 11333 ts_manager.cc:194] Registered new tserver with Master: 47f9a718ed0841dea18914e7a3322b05 (127.7.19.65:35035)
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
I20260928 12:26:18.431212 11333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:41621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:18.447777 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46053
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:18.528136 11528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:18.528385 11528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:18.528409 11528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:18.529577 11528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:18.529625 11528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:18.529651 11528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:18.531037 11528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:18.531207 11528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:18.531373 11528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:18.536280 11534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.536288 11533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.536298 11536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:18.536586 11528 server_base.cc:1034] running on GCE node
I20260928 12:26:18.536862 11528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:18.537094 11528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:18.538286 11528 hybrid_clock.cc:648] HybridClock initialized: now 1790598378538270 us; error 27 us; skew 500 ppm
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:18.540975 11528 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:18.541347 11528 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:18.541415 11528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:18.541549 11528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:18.542635 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "912cbd94d28349cd9ea47f65ea95e969"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "783c148c0a1f7a6d09059cee60ee56ef"
server_key_iv: "5bce4ed89c421696e1090d310b6e8422"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.543002 11528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "912cbd94d28349cd9ea47f65ea95e969"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "783c148c0a1f7a6d09059cee60ee56ef"
server_key_iv: "5bce4ed89c421696e1090d310b6e8422"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.544271 11528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:18.545037 11542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.545202 11528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.545295 11528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "912cbd94d28349cd9ea47f65ea95e969"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "783c148c0a1f7a6d09059cee60ee56ef"
server_key_iv: "5bce4ed89c421696e1090d310b6e8422"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.545375 11528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:18.561038 11528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:18.562882 11528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:18.563043 11528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:18.563257 11528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:18.563551 11528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:18.563604 11528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.563644 11528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:18.563673 11528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.569532 11528 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:37401
I20260928 12:26:18.569591 11655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:37401 every 8 connection(s)
I20260928 12:26:18.569871 11528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:18.572461 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11528
I20260928 12:26:18.572537 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:18.572753 7245 external_mini_cluster.cc:1428] Setting key 52163ea620355047232fb6c44ac47cc5
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:18.579672 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.570641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:34761 (local address 127.7.19.126:46053)
0928 12:26:18.570729 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.570732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.570893 (+ 161us) server_negotiation.cc:407] Connection header received
0928 12:26:18.571335 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.571336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.571348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.571409 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:18.572310 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.572995 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.574415 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.574596 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.574657 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.576850 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:18.576851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:18.576852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:18.576860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:18.578306 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.578713 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.578715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.578715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.578776 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.579040 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.579041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.579042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.579158 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:18.579232 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.579500 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.579583 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 12:26:18.579871 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.570601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46053 (local address 127.7.19.66:34761)
0928 12:26:18.570764 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.570797 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.571246 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.571471 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.571475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.571718 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:18.572195 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.572199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.573076 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.573077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.574325 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.574328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.574679 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.574680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.574831 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.575389 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:18.575397 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:18.576761 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:18.578408 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:18.578414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:18.578416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:18.578636 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:18.578850 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:18.578851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:18.578851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:18.578967 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:18.579222 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:18.579228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:18.579369 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.579478 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.579631 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 12:26:18.580422 11656 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46053
I20260928 12:26:18.580531 11656 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:18.580705 11656 heartbeater.cc:511] Master 127.7.19.126:46053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:26:18.581104 11333 ts_manager.cc:194] Registered new tserver with Master: 912cbd94d28349cd9ea47f65ea95e969 (127.7.19.66:37401)
I20260928 12:26:18.581537 11333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:34761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:18.599259 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46053
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:43193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:18.679603 11661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:18.679803 11661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:18.679826 11661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:18.680933 11661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:18.680967 11661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:18.680992 11661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:18.682379 11661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:18.682559 11661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:18.682715 11661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:18.687374 11666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.687353 11667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:18.687536 11661 server_base.cc:1034] running on GCE node
W20260928 12:26:18.687559 11669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:18.687776 11661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:18.688002 11661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:18.689159 11661 hybrid_clock.cc:648] HybridClock initialized: now 1790598378689143 us; error 26 us; skew 500 ppm
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:18.691609 11661 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:18.691920 11661 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:18.691989 11661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:18.692134 11661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:18.693176 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "06cdffeb67a54f788764954f2988dc30"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "5016dc8bb690094ec428f4fc92436ce6"
server_key_iv: "5bd6ea5b019bd739b928aee853dd8621"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.693539 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "06cdffeb67a54f788764954f2988dc30"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "5016dc8bb690094ec428f4fc92436ce6"
server_key_iv: "5bd6ea5b019bd739b928aee853dd8621"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.694772 11661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:18.695497 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.695680 11661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:18.695751 11661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "06cdffeb67a54f788764954f2988dc30"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "5016dc8bb690094ec428f4fc92436ce6"
server_key_iv: "5bd6ea5b019bd739b928aee853dd8621"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.695837 11661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:18.705304 11661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:18.707439 11661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:18.707587 11661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:18.707798 11661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:18.708082 11661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:18.708113 11661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.708134 11661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:18.708175 11661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.713786 11661 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:40167
I20260928 12:26:18.713856 11788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:40167 every 8 connection(s)
I20260928 12:26:18.714110 11661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:18 dist-test-slave-cb7x krb5kdc[11289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598378, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:18.723814 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11661
I20260928 12:26:18.723929 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:18.724182 7245 external_mini_cluster.cc:1428] Setting key 7a3cf6a19cba2364ee02ded6b86946cc
I20260928 12:26:18.726498 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.714837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:54905 (local address 127.7.19.126:46053)
0928 12:26:18.714905 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.714907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.715109 (+ 202us) server_negotiation.cc:407] Connection header received
0928 12:26:18.715667 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.715668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.715681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.715748 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:18.717199 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.718081 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.719760 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.720060 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.720149 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.723287 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:18.723289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:18.723289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:18.723313 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:18.725067 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.725496 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.725497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.725497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.725548 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.725812 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.725813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.725813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.725985 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:18.726039 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.726365 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.726435 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:26:18.726639 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.714833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46053 (local address 127.7.19.67:54905)
0928 12:26:18.714987 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.715020 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.715541 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.715839 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.715843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.716266 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:18.717100 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.717107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.718192 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.718193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.719675 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.719677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.720170 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.720171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.720313 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.721138 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:18.721148 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:18.723199 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:18.725141 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:18.725147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:18.725149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:18.725428 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:18.725636 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:18.725637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:18.725637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:18.725746 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:18.726066 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:18.726072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:18.726197 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.726300 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.726424 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260928 12:26:18.727139 11789 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46053
I20260928 12:26:18.727236 11789 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:18.727394 11789 heartbeater.cc:511] Master 127.7.19.126:46053 requested a full tablet report, sending...
I20260928 12:26:18.727814 11333 ts_manager.cc:194] Registered new tserver with Master: 06cdffeb67a54f788764954f2988dc30 (127.7.19.67:40167)
I20260928 12:26:18.728216 11333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:54905
I20260928 12:26:18.736004 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:26:18.743724 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.737408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.7.19.126:46053)
0928 12:26:18.737472 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.737474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.737657 (+ 183us) server_negotiation.cc:407] Connection header received
0928 12:26:18.737698 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.737699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.737708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.737758 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:18.738335 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.738817 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.739600 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.739878 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.739935 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.741401 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:18.741402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:18.741402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:18.741408 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:18.742729 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.743054 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.743055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.743055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.743108 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:18.743271 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:18.743271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:18.743272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:18.743438 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:18.743516 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.743602 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.743661 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:26:18.745541 11333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:18.747099 11333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:18.753599 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.749601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37401 (local address 127.0.0.1:33206)
0928 12:26:18.749764 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.749802 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.749875 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.750162 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.750165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.750167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:18.750240 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.750242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.751316 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.751317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.752544 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.752546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.753318 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.753319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.753393 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.753407 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.753480 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.753531 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260928 12:26:18.753611 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.749791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.7.19.66:37401)
0928 12:26:18.749923 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.749937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.749948 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:26:18.750013 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.750015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.750032 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.750325 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:18.750414 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.751218 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.752630 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.753235 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.753320 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.753389 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.753493 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.753531 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260928 12:26:18.753863 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.749601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:35035 (local address 127.0.0.1:45542)
0928 12:26:18.749917 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.749926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.749966 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.750234 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.750235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.750238 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:18.750292 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.750294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.751653 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.751654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.752737 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.752739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.753647 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.753648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.753711 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.753715 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.753745 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.753787 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260928 12:26:18.754043 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.750009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.7.19.67:40167)
0928 12:26:18.750170 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.750174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.750233 (+ 59us) server_negotiation.cc:407] Connection header received
0928 12:26:18.750384 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.750387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.750404 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.750486 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:18.750687 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.751736 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.752893 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.753544 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.753618 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.753673 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.753905 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.753953 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260928 12:26:18.754169 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.749804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40167 (local address 127.0.0.1:34300)
0928 12:26:18.750165 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.750176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.750288 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.750566 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.750567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.750569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:18.750607 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.750609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.751827 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.751828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.752804 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.752807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.753700 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.753701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.753758 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.753761 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.753805 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.753836 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":68,"threads_started":1}
I20260928 12:26:18.754199 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.749708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45542 (local address 127.7.19.65:35035)
0928 12:26:18.750025 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.750030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.750039 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:18.750100 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.750102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.750120 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.750368 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:18.750448 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.751541 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.752821 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.753545 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.753978 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.754047 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.754072 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.754120 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":75,"threads_started":1}
I20260928 12:26:18.754395 11590 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffe97e90d7a4b0dbfdb92806263674e (DEFAULT_TABLE table=test-table [id=cfe61460208d45a6a6ab9ac84dbb6a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:18.754782 11590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe97e90d7a4b0dbfdb92806263674e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:18.754802 11457 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffe97e90d7a4b0dbfdb92806263674e (DEFAULT_TABLE table=test-table [id=cfe61460208d45a6a6ab9ac84dbb6a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:18.755119 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe97e90d7a4b0dbfdb92806263674e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:18.757335 11723 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffe97e90d7a4b0dbfdb92806263674e (DEFAULT_TABLE table=test-table [id=cfe61460208d45a6a6ab9ac84dbb6a79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:18.757668 11723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe97e90d7a4b0dbfdb92806263674e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:18.758337 11806 tablet_bootstrap.cc:492] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Bootstrap starting.
I20260928 12:26:18.758628 11807 tablet_bootstrap.cc:492] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: Bootstrap starting.
I20260928 12:26:18.759043 11806 tablet_bootstrap.cc:654] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:18.759224 11807 tablet_bootstrap.cc:654] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:18.759727 11806 log.cc:824] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:18.759827 11807 log.cc:824] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:18.760468 11807 tablet_bootstrap.cc:492] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: No bootstrap required, opened a new log
I20260928 12:26:18.760497 11810 tablet_bootstrap.cc:492] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Bootstrap starting.
I20260928 12:26:18.760538 11807 ts_tablet_manager.cc:1397] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:18.760545 11806 tablet_bootstrap.cc:492] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: No bootstrap required, opened a new log
I20260928 12:26:18.760607 11806 ts_tablet_manager.cc:1397] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:18.761125 11810 tablet_bootstrap.cc:654] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:18.761686 11810 log.cc:824] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:18.762465 11806 raft_consensus.cc:348] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.762548 11810 tablet_bootstrap.cc:492] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: No bootstrap required, opened a new log
I20260928 12:26:18.762612 11810 ts_tablet_manager.cc:1397] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:18.762616 11806 raft_consensus.cc:374] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:18.762658 11806 raft_consensus.cc:729] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912cbd94d28349cd9ea47f65ea95e969, State: Initialized, Role: FOLLOWER
I20260928 12:26:18.762764 11806 consensus_queue.cc:260] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.762705 11807 raft_consensus.cc:348] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.762852 11807 raft_consensus.cc:374] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:18.762895 11807 raft_consensus.cc:729] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f9a718ed0841dea18914e7a3322b05, State: Initialized, Role: FOLLOWER
I20260928 12:26:18.762995 11807 consensus_queue.cc:260] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.763043 11806 ts_tablet_manager.cc:1428] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:26:18.763188 11656 heartbeater.cc:503] Master 127.7.19.126:46053 was elected leader, sending a full tablet report...
I20260928 12:26:18.763252 11807 ts_tablet_manager.cc:1428] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:26:18.763371 11523 heartbeater.cc:503] Master 127.7.19.126:46053 was elected leader, sending a full tablet report...
I20260928 12:26:18.764612 11810 raft_consensus.cc:348] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.764807 11810 raft_consensus.cc:374] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:18.764863 11810 raft_consensus.cc:729] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cdffeb67a54f788764954f2988dc30, State: Initialized, Role: FOLLOWER
I20260928 12:26:18.764997 11810 consensus_queue.cc:260] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.765259 11810 ts_tablet_manager.cc:1428] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:26:18.765285 11789 heartbeater.cc:503] Master 127.7.19.126:46053 was elected leader, sending a full tablet report...
I20260928 12:26:18.780927 11813 raft_consensus.cc:482] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:18.781045 11813 raft_consensus.cc:504] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.781317 11813 leader_election.cc:288] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912cbd94d28349cd9ea47f65ea95e969 (127.7.19.66:37401), 06cdffeb67a54f788764954f2988dc30 (127.7.19.67:40167)
I20260928 12:26:18.785228 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.781737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:39891 (local address 127.7.19.67:40167)
0928 12:26:18.781778 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.781781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.781892 (+ 111us) server_negotiation.cc:407] Connection header received
0928 12:26:18.781946 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.781948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.781961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.782013 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:18.782163 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.782862 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.784255 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.784894 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.784955 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.784964 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.785117 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.785156 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:18.785228 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.781602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:59307 (local address 127.7.19.66:37401)
0928 12:26:18.781671 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.781673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.781685 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:26:18.781761 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.781762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.781774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.781829 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:18.781981 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.782769 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.784255 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.784849 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.784928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.784941 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.785117 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.785157 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:26:18.785255 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.781535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37401 (local address 127.7.19.65:59307)
0928 12:26:18.781632 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.781645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.781703 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.781839 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.781840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.781842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:18.781911 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.781913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.782850 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.782850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.784162 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.784164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.784952 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.784953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.785043 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.785045 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.785099 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.785158 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":52}
I20260928 12:26:18.785260 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.781670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40167 (local address 127.7.19.65:39891)
0928 12:26:18.781830 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.781840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:18.781896 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:18.782031 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:18.782031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:18.782033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:18.782095 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.782097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.782936 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.782937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.784162 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:18.784164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.784957 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.784958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:18.785033 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.785038 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:18.785098 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 12:26:18.785158 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":14,"thread_start_us":112,"threads_started":1}
I20260928 12:26:18.785727 11610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe97e90d7a4b0dbfdb92806263674e" candidate_uuid: "47f9a718ed0841dea18914e7a3322b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912cbd94d28349cd9ea47f65ea95e969" is_pre_election: true
I20260928 12:26:18.785738 11743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe97e90d7a4b0dbfdb92806263674e" candidate_uuid: "47f9a718ed0841dea18914e7a3322b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06cdffeb67a54f788764954f2988dc30" is_pre_election: true
I20260928 12:26:18.785887 11610 raft_consensus.cc:2454] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f9a718ed0841dea18914e7a3322b05 in term 0.
I20260928 12:26:18.785892 11743 raft_consensus.cc:2454] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f9a718ed0841dea18914e7a3322b05 in term 0.
I20260928 12:26:18.786126 11413 leader_election.cc:302] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f9a718ed0841dea18914e7a3322b05, 912cbd94d28349cd9ea47f65ea95e969; no voters:
I20260928 12:26:18.786281 11813 raft_consensus.cc:2781] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:18.786355 11813 raft_consensus.cc:482] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:18.786391 11813 raft_consensus.cc:3037] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:18.787078 11813 raft_consensus.cc:504] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.787214 11813 leader_election.cc:288] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [CANDIDATE]: Term 1 election: Requested vote from peers 912cbd94d28349cd9ea47f65ea95e969 (127.7.19.66:37401), 06cdffeb67a54f788764954f2988dc30 (127.7.19.67:40167)
I20260928 12:26:18.787375 11610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe97e90d7a4b0dbfdb92806263674e" candidate_uuid: "47f9a718ed0841dea18914e7a3322b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912cbd94d28349cd9ea47f65ea95e969"
I20260928 12:26:18.787462 11743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe97e90d7a4b0dbfdb92806263674e" candidate_uuid: "47f9a718ed0841dea18914e7a3322b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06cdffeb67a54f788764954f2988dc30"
I20260928 12:26:18.787470 11610 raft_consensus.cc:3037] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:18.787542 11743 raft_consensus.cc:3037] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:18.788147 11610 raft_consensus.cc:2454] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f9a718ed0841dea18914e7a3322b05 in term 1.
I20260928 12:26:18.788305 11743 raft_consensus.cc:2454] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f9a718ed0841dea18914e7a3322b05 in term 1.
I20260928 12:26:18.788363 11413 leader_election.cc:302] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f9a718ed0841dea18914e7a3322b05, 912cbd94d28349cd9ea47f65ea95e969; no voters:
I20260928 12:26:18.788475 11813 raft_consensus.cc:2781] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:18.788619 11813 raft_consensus.cc:686] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 1 LEADER]: Becoming Leader. State: Replica: 47f9a718ed0841dea18914e7a3322b05, State: Running, Role: LEADER
I20260928 12:26:18.788746 11813 consensus_queue.cc:237] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } }
I20260928 12:26:18.789570 11332 catalog_manager.cc:5495] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f9a718ed0841dea18914e7a3322b05 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "47f9a718ed0841dea18914e7a3322b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f9a718ed0841dea18914e7a3322b05" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:18.795224 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.793031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.7.19.65:35035)
0928 12:26:18.793085 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.793087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.793095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:18.793155 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.793155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.793170 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.793221 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:18.793390 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.793831 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.794566 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:18.794743 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:18.794801 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:18.794958 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:18.795047 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:18.795110 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.795142 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.795159 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:26:18.796875 11743 raft_consensus.cc:1261] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 1 FOLLOWER]: Refusing update from remote peer 47f9a718ed0841dea18914e7a3322b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:18.796921 11610 raft_consensus.cc:1261] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 1 FOLLOWER]: Refusing update from remote peer 47f9a718ed0841dea18914e7a3322b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:18.797194 11816 consensus_queue.cc:1035] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06cdffeb67a54f788764954f2988dc30" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:18.797350 11813 consensus_queue.cc:1035] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912cbd94d28349cd9ea47f65ea95e969" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:18.799914 11821 mvcc.cc:205] Tried to move back new op lower bound from 7334290959549747200 to 7334290959519248384. Current Snapshot: MvccSnapshot[applied={T|T < 7334290959549747200}]
I20260928 12:26:18.801847 11332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:18.801941 11332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:18.802670 11332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878e812714ad46ba9f241eb7d0c0e212: Sending DeleteTablet for 3 replicas of tablet 3ffe97e90d7a4b0dbfdb92806263674e
I20260928 12:26:18.802978 11590 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffe97e90d7a4b0dbfdb92806263674e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:18 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:33206
I20260928 12:26:18.803031 11457 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffe97e90d7a4b0dbfdb92806263674e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:18 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:45542
I20260928 12:26:18.803074 11723 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffe97e90d7a4b0dbfdb92806263674e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:18 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:34300
I20260928 12:26:18.803241 11830 tablet_replica.cc:330] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: stopping tablet replica
I20260928 12:26:18.803257 11831 tablet_replica.cc:330] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: stopping tablet replica
I20260928 12:26:18.803330 11830 raft_consensus.cc:2229] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:18.803340 11831 raft_consensus.cc:2229] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:18.803435 11830 raft_consensus.cc:2258] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:18.803428 11831 raft_consensus.cc:2258] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:18.803637 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11395
I20260928 12:26:18.803870 11830 ts_tablet_manager.cc:1905] T 3ffe97e90d7a4b0dbfdb92806263674e P 47f9a718ed0841dea18914e7a3322b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:18.804136 11829 tablet_replica.cc:330] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: stopping tablet replica
I20260928 12:26:18.804235 11829 raft_consensus.cc:2229] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:18.804299 11829 raft_consensus.cc:2258] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:18.806103 11829 ts_tablet_manager.cc:1905] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:18.806114 11831 ts_tablet_manager.cc:1905] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:18.807824 11829 ts_tablet_manager.cc:1918] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:18.807915 11829 log.cc:1196] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/3ffe97e90d7a4b0dbfdb92806263674e
I20260928 12:26:18.808095 11831 ts_tablet_manager.cc:1918] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:18.808167 11831 log.cc:1196] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/3ffe97e90d7a4b0dbfdb92806263674e
I20260928 12:26:18.808231 11829 ts_tablet_manager.cc:1939] T 3ffe97e90d7a4b0dbfdb92806263674e P 912cbd94d28349cd9ea47f65ea95e969: Deleting consensus metadata
I20260928 12:26:18.808425 11831 ts_tablet_manager.cc:1939] T 3ffe97e90d7a4b0dbfdb92806263674e P 06cdffeb67a54f788764954f2988dc30: Deleting consensus metadata
I20260928 12:26:18.808813 11320 catalog_manager.cc:4841] TS 912cbd94d28349cd9ea47f65ea95e969 (127.7.19.66:37401): tablet 3ffe97e90d7a4b0dbfdb92806263674e (table test-table [id=cfe61460208d45a6a6ab9ac84dbb6a79]) successfully deleted
I20260928 12:26:18.808980 11320 catalog_manager.cc:4841] TS 06cdffeb67a54f788764954f2988dc30 (127.7.19.67:40167): tablet 3ffe97e90d7a4b0dbfdb92806263674e (table test-table [id=cfe61460208d45a6a6ab9ac84dbb6a79]) successfully deleted
W20260928 12:26:18.809711 11318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:35035 (error 108)
I20260928 12:26:18.810204 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11528
I20260928 12:26:18.810225 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.810100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:35035 (local address 127.0.0.1:45560)
0928 12:26:18.810152 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:18.810188 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:35035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:26:18.811472 11318 catalog_manager.cc:4584] TS 47f9a718ed0841dea18914e7a3322b05 (127.7.19.65:35035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ffe97e90d7a4b0dbfdb92806263674e: Network error: Client connection negotiation failed: client connection to 127.7.19.65:35035: connect: Connection refused (error 111)
I20260928 12:26:18.815124 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11661
I20260928 12:26:18.819820 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11302
2026-09-28T12:26:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1895 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T12:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:18Z Disabled control of system clock
I20260928 12:26:18.846940 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43807
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:26:18.925156 11836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:18.925349 11836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:18.925371 11836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:18.926406 11836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:18.926446 11836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:18.926460 11836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:18.926471 11836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:18.926483 11836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:18.927704 11836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:43807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:18.927875 11836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:18.928093 11836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:18.932828 11844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.932828 11842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:18.932905 11841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:18.933048 11836 server_base.cc:1034] running on GCE node
I20260928 12:26:18.933256 11836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:18.933511 11836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:18.934646 11836 hybrid_clock.cc:648] HybridClock initialized: now 1790598378934629 us; error 40 us; skew 500 ppm
I20260928 12:26:18.935835 11836 webserver.cc:460] Webserver started at http://127.7.19.126:46261/ using document root <none> and password file <none>
I20260928 12:26:18.936062 11836 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:18.936126 11836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:18.936268 11836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:18.937354 11836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "794dd06b36214c7c9799df6fb2e9ec02"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "7105ea8ffa0c99262323969918e9a9c2"
server_key_iv: "0997b3bd5debf0ecf9647a756da3da95"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.937704 11836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "794dd06b36214c7c9799df6fb2e9ec02"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "7105ea8ffa0c99262323969918e9a9c2"
server_key_iv: "0997b3bd5debf0ecf9647a756da3da95"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.938997 11836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:18.939678 11850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:18.939847 11836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:18.939944 11836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "794dd06b36214c7c9799df6fb2e9ec02"
format_stamp: "Formatted at 2026-09-28 12:26:18 on dist-test-slave-cb7x"
server_key: "7105ea8ffa0c99262323969918e9a9c2"
server_key_iv: "0997b3bd5debf0ecf9647a756da3da95"
server_key_version: "encryptionkey@0"
I20260928 12:26:18.940012 11836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:18.943290 11836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:18.943538 11836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:18.943668 11836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:18.947326 11836 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:43807
I20260928 12:26:18.947357 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:43807 every 8 connection(s)
I20260928 12:26:18.947662 11836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:18.948244 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:18.950598 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11836
I20260928 12:26:18.950702 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:18.950875 7245 external_mini_cluster.cc:1428] Setting key 5b2fc0a5d026b30c0909bcb332c383e8
I20260928 12:26:18.950870 11903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02: Bootstrap starting.
I20260928 12:26:18.951586 11903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:18.951901 11903 log.cc:824] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:18.952618 11903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02: No bootstrap required, opened a new log
I20260928 12:26:18.953461 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:18.951369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.7.19.126:43807)
0928 12:26:18.951562 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:18.951572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:18.951604 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:26:18.952172 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:18.952178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:18.952213 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:18.952351 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:18.953025 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:18.953028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:18.953029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:18.953033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:18.953077 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:18.953091 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:18.953178 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:18.953211 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:26:18.953263 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260928 12:26:18.954461 11903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd06b36214c7c9799df6fb2e9ec02" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43807 } }
I20260928 12:26:18.954583 11903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:18.954605 11903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794dd06b36214c7c9799df6fb2e9ec02, State: Initialized, Role: FOLLOWER
I20260928 12:26:18.954742 11903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd06b36214c7c9799df6fb2e9ec02" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43807 } }
I20260928 12:26:18.954823 11903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:18.954872 11903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:18.954921 11903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:18.955694 11903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd06b36214c7c9799df6fb2e9ec02" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43807 } }
I20260928 12:26:18.955802 11903 leader_election.cc:302] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794dd06b36214c7c9799df6fb2e9ec02; no voters:
I20260928 12:26:18.955955 11903 leader_election.cc:288] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:18.956012 11908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:18.956146 11908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [term 1 LEADER]: Becoming Leader. State: Replica: 794dd06b36214c7c9799df6fb2e9ec02, State: Running, Role: LEADER
I20260928 12:26:18.956195 11903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:18.956238 11908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd06b36214c7c9799df6fb2e9ec02" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43807 } }
I20260928 12:26:18.956578 11909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "794dd06b36214c7c9799df6fb2e9ec02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd06b36214c7c9799df6fb2e9ec02" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43807 } } }
I20260928 12:26:18.956578 11910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 794dd06b36214c7c9799df6fb2e9ec02. Latest consensus state: current_term: 1 leader_uuid: "794dd06b36214c7c9799df6fb2e9ec02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd06b36214c7c9799df6fb2e9ec02" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43807 } } }
I20260928 12:26:18.956715 11909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:18.956723 11910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:18.956981 11916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:18.957492 11916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:18.958855 11916 catalog_manager.cc:1347] Generated new cluster ID: ecf847cd2a944820af053dc0c9f4f62d
I20260928 12:26:18.958921 11916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:18.963608 11916 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:18.963742 11916 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:18.969197 11916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02: Generated new TSK 0
I20260928 12:26:18.969339 11916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:18.970394 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43807
--builtin_ntp_servers=127.7.19.84:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:26:19.050078 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:19.050304 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:19.050339 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:19.051474 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:19.051537 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:19.051577 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:19.052968 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:19.053208 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:19.053484 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:19.059280 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.059312 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.059468 11927 server_base.cc:1034] running on GCE node
W20260928 12:26:19.059476 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.059823 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:19.060110 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:19.061259 11927 hybrid_clock.cc:648] HybridClock initialized: now 1790598379061243 us; error 33 us; skew 500 ppm
I20260928 12:26:19.062451 11927 webserver.cc:460] Webserver started at http://127.7.19.65:38879/ using document root <none> and password file <none>
I20260928 12:26:19.062683 11927 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:19.062752 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:19.062901 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:19.063925 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "de00c3856386ced07f6e15cd3cec386b"
server_key_iv: "68523706f306c622d6c23fa66845b382"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.064306 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "de00c3856386ced07f6e15cd3cec386b"
server_key_iv: "68523706f306c622d6c23fa66845b382"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.065656 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:19.066388 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.066603 11927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:19.066696 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "de00c3856386ced07f6e15cd3cec386b"
server_key_iv: "68523706f306c622d6c23fa66845b382"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.066761 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:19.071229 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:19.071492 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:19.071616 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:19.071822 11927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:19.072122 11927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:19.072161 11927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.072187 11927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:19.072218 11927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.079282 11927 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:35077
I20260928 12:26:19.079313 12054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:35077 every 8 connection(s)
I20260928 12:26:19.079648 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:19.082618 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.080306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:53381 (local address 127.7.19.126:43807)
0928 12:26:19.080376 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.080378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.080627 (+ 249us) server_negotiation.cc:407] Connection header received
0928 12:26:19.081148 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.081149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.081155 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.081206 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.082366 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.082367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.082367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.082372 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.082387 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.082391 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.082443 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.082536 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.082551 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:26:19.082809 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.080299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43807 (local address 127.7.19.65:53381)
0928 12:26:19.080496 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.080533 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.081039 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.081271 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.081275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.081284 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.082217 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.082218 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.082244 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.082272 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.082445 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.082450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.082453 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.082547 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.082605 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260928 12:26:19.083249 12055 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43807
I20260928 12:26:19.083352 12055 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:19.083490 12055 heartbeater.cc:511] Master 127.7.19.126:43807 requested a full tablet report, sending...
I20260928 12:26:19.083937 11867 ts_manager.cc:194] Registered new tserver with Master: 8bdcdab0cbd6410fb72f3b00d7691aaa (127.7.19.65:35077)
I20260928 12:26:19.085054 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 11927
I20260928 12:26:19.085166 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:19.085333 7245 external_mini_cluster.cc:1428] Setting key f42ae9af49ace4fa55443fe716c61241
I20260928 12:26:19.086169 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43807
--builtin_ntp_servers=127.7.19.84:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:26:19.165529 12058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:19.165712 12058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:19.165736 12058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:19.166852 12058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:19.166891 12058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:19.166916 12058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:19.168205 12058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:19.168435 12058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:19.168768 12058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:19.173905 12058 server_base.cc:1034] running on GCE node
W20260928 12:26:19.173905 12066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.174048 12063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.173929 12064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.174461 12058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:19.174723 12058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:19.175906 12058 hybrid_clock.cc:648] HybridClock initialized: now 1790598379175891 us; error 45 us; skew 500 ppm
I20260928 12:26:19.177400 12058 webserver.cc:460] Webserver started at http://127.7.19.66:44625/ using document root <none> and password file <none>
I20260928 12:26:19.177657 12058 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:19.177737 12058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:19.177906 12058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:19.179147 12058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "ee9239ad9ddf4701af431ec230a74b9d"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "b38a61c20425547f86cd259e21e0cb51"
server_key_iv: "278c0d6f140e8515a7b7b6199054fdaa"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.179553 12058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "ee9239ad9ddf4701af431ec230a74b9d"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "b38a61c20425547f86cd259e21e0cb51"
server_key_iv: "278c0d6f140e8515a7b7b6199054fdaa"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.181015 12058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:19.181795 12072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.181984 12058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:19.182083 12058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "ee9239ad9ddf4701af431ec230a74b9d"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "b38a61c20425547f86cd259e21e0cb51"
server_key_iv: "278c0d6f140e8515a7b7b6199054fdaa"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.182161 12058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:19.185470 12058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:19.185722 12058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:19.185868 12058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:19.186067 12058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:19.186363 12058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:19.186416 12058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.186458 12058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:19.186477 12058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.192134 12058 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:38169
I20260928 12:26:19.192171 12185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:38169 every 8 connection(s)
I20260928 12:26:19.192489 12058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:19.195390 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.193265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:55959 (local address 127.7.19.126:43807)
0928 12:26:19.193343 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.193345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.193558 (+ 213us) server_negotiation.cc:407] Connection header received
0928 12:26:19.194002 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.194002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.194008 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.194058 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.195228 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.195228 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.195229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.195231 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.195241 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.195245 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.195286 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.195332 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.195342 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:26:19.195602 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.193232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43807 (local address 127.7.19.66:55959)
0928 12:26:19.193461 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.193491 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.193916 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.194141 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.194145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.194155 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.195105 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.195106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.195132 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.195159 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.195286 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.195288 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.195290 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.195338 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.195382 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":130,"threads_started":1}
I20260928 12:26:19.196043 12186 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43807
I20260928 12:26:19.196130 12186 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:19.196228 12186 heartbeater.cc:511] Master 127.7.19.126:43807 requested a full tablet report, sending...
I20260928 12:26:19.196553 11867 ts_manager.cc:194] Registered new tserver with Master: ee9239ad9ddf4701af431ec230a74b9d (127.7.19.66:38169)
I20260928 12:26:19.200767 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12058
I20260928 12:26:19.200860 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:19.201041 7245 external_mini_cluster.cc:1428] Setting key 99a04be82e0f7e55ace70fb40bcae17b
I20260928 12:26:19.201923 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43807
--builtin_ntp_servers=127.7.19.84:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 12:26:19.282907 12189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:19.283118 12189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:19.283142 12189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:19.284243 12189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:19.284281 12189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:19.284307 12189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:19.285774 12189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:19.286017 12189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:19.286303 12189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:19.291332 12189 server_base.cc:1034] running on GCE node
W20260928 12:26:19.291311 12194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.291298 12197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.291290 12195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.291651 12189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:19.291905 12189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:19.293095 12189 hybrid_clock.cc:648] HybridClock initialized: now 1790598379293084 us; error 56 us; skew 500 ppm
I20260928 12:26:19.294333 12189 webserver.cc:460] Webserver started at http://127.7.19.67:42089/ using document root <none> and password file <none>
I20260928 12:26:19.294596 12189 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:19.294665 12189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:19.294835 12189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:19.295877 12189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "e80c12b0645d4bd988d857ded26c5b86"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "17ca6c1435dddd9766f68915a8c0f331"
server_key_iv: "ef539d66bafd2d3518e79717c2007273"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.296249 12189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "e80c12b0645d4bd988d857ded26c5b86"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "17ca6c1435dddd9766f68915a8c0f331"
server_key_iv: "ef539d66bafd2d3518e79717c2007273"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.297612 12189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:19.298352 12203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.298527 12189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:19.298631 12189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "e80c12b0645d4bd988d857ded26c5b86"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "17ca6c1435dddd9766f68915a8c0f331"
server_key_iv: "ef539d66bafd2d3518e79717c2007273"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.298703 12189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:19.302098 12189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:19.302348 12189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:19.302486 12189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:19.302676 12189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:19.302971 12189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:19.303021 12189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.303058 12189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:19.303086 12189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.308871 12189 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:35429
I20260928 12:26:19.308915 12316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:35429 every 8 connection(s)
I20260928 12:26:19.309230 12189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:19.312153 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.309900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:59387 (local address 127.7.19.126:43807)
0928 12:26:19.309944 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.309946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.310166 (+ 220us) server_negotiation.cc:407] Connection header received
0928 12:26:19.310658 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.310659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.310667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.310713 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.311906 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.311908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.311908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.311914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.311928 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.311933 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.311985 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.312071 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.312085 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:26:19.312379 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.309885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43807 (local address 127.7.19.67:59387)
0928 12:26:19.310052 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.310083 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.310552 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.310790 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.310794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.310804 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.311757 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.311758 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.311784 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.311811 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.311982 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.311987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.311990 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.312066 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.312125 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 12:26:19.312865 12317 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43807
I20260928 12:26:19.312973 12317 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:19.313081 12317 heartbeater.cc:511] Master 127.7.19.126:43807 requested a full tablet report, sending...
I20260928 12:26:19.313414 11867 ts_manager.cc:194] Registered new tserver with Master: e80c12b0645d4bd988d857ded26c5b86 (127.7.19.67:35429)
I20260928 12:26:19.316591 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12189
I20260928 12:26:19.316727 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:19.316900 7245 external_mini_cluster.cc:1428] Setting key 3de0463e1ff7f7bd4cdca33f82ead91b
I20260928 12:26:19.317694 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:26:19.319918 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.318885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.7.19.126:43807)
0928 12:26:19.318927 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.318929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.319101 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:26:19.319136 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.319136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.319140 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.319186 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.319718 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.319719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.319719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.319722 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.319733 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.319737 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.319779 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.319810 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.319819 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:26:19.321473 11867 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:19.322903 11867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:19.326251 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.324896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:38169 (local address 127.0.0.1:46370)
0928 12:26:19.325012 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.325040 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.325104 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.325330 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.325344 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.325350 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.325867 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.325868 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.325883 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.325904 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.326130 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.326141 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.326144 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.326195 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.326206 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":80,"threads_started":1}
I20260928 12:26:19.326354 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.325000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.7.19.66:38169)
0928 12:26:19.325170 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.325175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.325184 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:19.325212 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.325214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.325224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.325536 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.325972 (+ 436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.325977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.325977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.325982 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.326016 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.326040 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.326135 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.326224 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.326243 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260928 12:26:19.327057 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.325286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:35077 (local address 127.0.0.1:57994)
0928 12:26:19.325584 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.325594 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.325635 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.326159 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.326160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.326162 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.326659 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.326659 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.326666 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.326677 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.326937 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.326938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.326939 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.326979 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.326993 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":67,"threads_started":1}
I20260928 12:26:19.327219 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.325112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:35429 (local address 127.0.0.1:37542)
0928 12:26:19.325227 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.325236 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.325270 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.326489 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.326490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.326493 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.326853 (+ 360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.326854 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.326861 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.326876 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.327091 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.327092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.327092 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.327142 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.327156 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260928 12:26:19.327232 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.325469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.7.19.67:35429)
0928 12:26:19.326364 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.326368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.326377 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:19.326400 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.326402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.326410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.326489 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.326973 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.326980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.326980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.326984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.327019 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.327041 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.327097 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.327146 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.327160 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":74,"threads_started":1}
I20260928 12:26:19.327235 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.325393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.7.19.65:35077)
0928 12:26:19.325746 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.325750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.325759 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:19.325796 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.325798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.325806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.325888 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.326761 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.326768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.326768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.326772 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.326809 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.326859 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.326914 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.327141 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.327160 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":93,"threads_started":1}
I20260928 12:26:19.327517 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3ce093d7ec4bf786faf9a411365cf6 (DEFAULT_TABLE table=test-table [id=12bb594078a648ac8834be329f21e3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:19.327862 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3ce093d7ec4bf786faf9a411365cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:19.328060 12251 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3ce093d7ec4bf786faf9a411365cf6 (DEFAULT_TABLE table=test-table [id=12bb594078a648ac8834be329f21e3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:19.328373 12251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3ce093d7ec4bf786faf9a411365cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:19.329264 11989 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3ce093d7ec4bf786faf9a411365cf6 (DEFAULT_TABLE table=test-table [id=12bb594078a648ac8834be329f21e3d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:19.329576 11989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3ce093d7ec4bf786faf9a411365cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:19.331373 12331 tablet_bootstrap.cc:492] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Bootstrap starting.
I20260928 12:26:19.331728 12332 tablet_bootstrap.cc:492] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Bootstrap starting.
I20260928 12:26:19.332090 12331 tablet_bootstrap.cc:654] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:19.332331 12332 tablet_bootstrap.cc:654] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:19.332423 12331 log.cc:824] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:19.332667 12332 log.cc:824] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:19.332830 12333 tablet_bootstrap.cc:492] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa: Bootstrap starting.
I20260928 12:26:19.333293 12331 tablet_bootstrap.cc:492] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: No bootstrap required, opened a new log
I20260928 12:26:19.333323 12332 tablet_bootstrap.cc:492] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: No bootstrap required, opened a new log
I20260928 12:26:19.333364 12331 ts_tablet_manager.cc:1397] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:19.333400 12332 ts_tablet_manager.cc:1397] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:19.333484 12333 tablet_bootstrap.cc:654] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:19.333789 12333 log.cc:824] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:19.334532 12333 tablet_bootstrap.cc:492] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa: No bootstrap required, opened a new log
I20260928 12:26:19.334597 12333 ts_tablet_manager.cc:1397] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:19.335638 12332 raft_consensus.cc:348] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.335788 12332 raft_consensus.cc:374] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:19.335829 12332 raft_consensus.cc:729] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80c12b0645d4bd988d857ded26c5b86, State: Initialized, Role: FOLLOWER
I20260928 12:26:19.335950 12332 consensus_queue.cc:260] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.335985 12331 raft_consensus.cc:348] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.336161 12331 raft_consensus.cc:374] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:19.336205 12331 raft_consensus.cc:729] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9239ad9ddf4701af431ec230a74b9d, State: Initialized, Role: FOLLOWER
I20260928 12:26:19.336215 12332 ts_tablet_manager.cc:1428] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:26:19.336355 12317 heartbeater.cc:503] Master 127.7.19.126:43807 was elected leader, sending a full tablet report...
I20260928 12:26:19.336336 12331 consensus_queue.cc:260] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.336598 12331 ts_tablet_manager.cc:1428] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:26:19.336619 12186 heartbeater.cc:503] Master 127.7.19.126:43807 was elected leader, sending a full tablet report...
I20260928 12:26:19.337136 12333 raft_consensus.cc:348] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.337309 12333 raft_consensus.cc:374] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:19.337373 12333 raft_consensus.cc:729] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bdcdab0cbd6410fb72f3b00d7691aaa, State: Initialized, Role: FOLLOWER
I20260928 12:26:19.337484 12333 consensus_queue.cc:260] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.337728 12333 ts_tablet_manager.cc:1428] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:26:19.337865 12055 heartbeater.cc:503] Master 127.7.19.126:43807 was elected leader, sending a full tablet report...
I20260928 12:26:19.355751 12339 raft_consensus.cc:482] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:19.355845 12339 raft_consensus.cc:504] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.356134 12339 leader_election.cc:288] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9239ad9ddf4701af431ec230a74b9d (127.7.19.66:38169), e80c12b0645d4bd988d857ded26c5b86 (127.7.19.67:35429)
I20260928 12:26:19.357736 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.356514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:35821 (local address 127.7.19.67:35429)
0928 12:26:19.356569 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.356571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.356580 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:19.356631 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.356632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.356637 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.356734 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.357439 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.357440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.357441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.357446 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.357491 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.357503 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.357531 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.357651 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.357664 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:26:19.357754 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.356383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:35429 (local address 127.7.19.65:35821)
0928 12:26:19.356530 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.356539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.356578 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.356740 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.356741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.356744 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.357319 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.357320 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.357328 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.357345 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.357569 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.357570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.357570 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.357629 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.357645 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260928 12:26:19.357805 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.356592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:48441 (local address 127.7.19.66:38169)
0928 12:26:19.356645 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.356647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.356662 (+ 15us) server_negotiation.cc:407] Connection header received
0928 12:26:19.356738 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.356739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.356744 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.356805 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.357562 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.357563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.357563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.357568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.357582 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.357585 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.357658 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.357721 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.357734 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:26:19.358196 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.356286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:38169 (local address 127.7.19.65:48441)
0928 12:26:19.356324 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.356335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.356392 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.356813 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.356814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.356816 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:19.357459 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:19.357460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:19.357466 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:19.357477 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:19.357653 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:19.357654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:19.357654 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 12:26:19.357718 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:26:19.357732 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 12:26:19.358354 12271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3ce093d7ec4bf786faf9a411365cf6" candidate_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80c12b0645d4bd988d857ded26c5b86" is_pre_election: true
I20260928 12:26:19.358530 12271 raft_consensus.cc:2454] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bdcdab0cbd6410fb72f3b00d7691aaa in term 0.
I20260928 12:26:19.358559 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3ce093d7ec4bf786faf9a411365cf6" candidate_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9239ad9ddf4701af431ec230a74b9d" is_pre_election: true
I20260928 12:26:19.358691 12140 raft_consensus.cc:2454] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bdcdab0cbd6410fb72f3b00d7691aaa in term 0.
I20260928 12:26:19.358776 11943 leader_election.cc:302] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bdcdab0cbd6410fb72f3b00d7691aaa, e80c12b0645d4bd988d857ded26c5b86; no voters:
I20260928 12:26:19.359057 12339 raft_consensus.cc:2781] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:19.359129 12339 raft_consensus.cc:482] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:19.359146 12339 raft_consensus.cc:3037] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:19.359895 12339 raft_consensus.cc:504] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.360021 12339 leader_election.cc:288] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [CANDIDATE]: Term 1 election: Requested vote from peers ee9239ad9ddf4701af431ec230a74b9d (127.7.19.66:38169), e80c12b0645d4bd988d857ded26c5b86 (127.7.19.67:35429)
I20260928 12:26:19.360203 12140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3ce093d7ec4bf786faf9a411365cf6" candidate_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9239ad9ddf4701af431ec230a74b9d"
I20260928 12:26:19.360215 12271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3ce093d7ec4bf786faf9a411365cf6" candidate_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e80c12b0645d4bd988d857ded26c5b86"
I20260928 12:26:19.360282 12271 raft_consensus.cc:3037] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:19.360283 12140 raft_consensus.cc:3037] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:19.361178 12271 raft_consensus.cc:2454] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bdcdab0cbd6410fb72f3b00d7691aaa in term 1.
I20260928 12:26:19.361178 12140 raft_consensus.cc:2454] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bdcdab0cbd6410fb72f3b00d7691aaa in term 1.
I20260928 12:26:19.361369 11943 leader_election.cc:302] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bdcdab0cbd6410fb72f3b00d7691aaa, ee9239ad9ddf4701af431ec230a74b9d; no voters:
I20260928 12:26:19.361446 12339 raft_consensus.cc:2781] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:19.361599 12339 raft_consensus.cc:686] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [term 1 LEADER]: Becoming Leader. State: Replica: 8bdcdab0cbd6410fb72f3b00d7691aaa, State: Running, Role: LEADER
I20260928 12:26:19.361697 12339 consensus_queue.cc:237] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } }
I20260928 12:26:19.362502 11867 catalog_manager.cc:5495] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bdcdab0cbd6410fb72f3b00d7691aaa (127.7.19.65). New cstate: current_term: 1 leader_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bdcdab0cbd6410fb72f3b00d7691aaa" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 35077 } health_report { overall_health: HEALTHY } } }
I20260928 12:26:19.374708 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.373730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.7.19.65:35077)
0928 12:26:19.373767 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.373769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.373777 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:19.373860 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.373860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.373878 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.373941 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.374471 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.374472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.374472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.374476 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.374493 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.374503 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.374566 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.374624 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.374638 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:26:19.376363 12140 raft_consensus.cc:1261] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 1 FOLLOWER]: Refusing update from remote peer 8bdcdab0cbd6410fb72f3b00d7691aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:19.376380 12271 raft_consensus.cc:1261] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 1 FOLLOWER]: Refusing update from remote peer 8bdcdab0cbd6410fb72f3b00d7691aaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:19.376709 12339 consensus_queue.cc:1035] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9239ad9ddf4701af431ec230a74b9d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:19.376850 12341 consensus_queue.cc:1035] T 2c3ce093d7ec4bf786faf9a411365cf6 P 8bdcdab0cbd6410fb72f3b00d7691aaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e80c12b0645d4bd988d857ded26c5b86" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:19.378971 12344 mvcc.cc:205] Tried to move back new op lower bound from 7334290961922912256 to 7334290961866133504. Current Snapshot: MvccSnapshot[applied={T|T < 7334290961922912256}]
I20260928 12:26:19.379660 12345 mvcc.cc:205] Tried to move back new op lower bound from 7334290961922912256 to 7334290961866133504. Current Snapshot: MvccSnapshot[applied={T|T < 7334290961922912256}]
I20260928 12:26:19.379894 12346 mvcc.cc:205] Tried to move back new op lower bound from 7334290961922912256 to 7334290961866133504. Current Snapshot: MvccSnapshot[applied={T|T < 7334290961922912256}]
I20260928 12:26:19.381331 11866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:19.381417 11866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:19.382138 11866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794dd06b36214c7c9799df6fb2e9ec02: Sending DeleteTablet for 3 replicas of tablet 2c3ce093d7ec4bf786faf9a411365cf6
I20260928 12:26:19.382515 11989 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3ce093d7ec4bf786faf9a411365cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:19 UTC) from {username='slave'} at 127.0.0.1:57994
I20260928 12:26:19.382521 12251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3ce093d7ec4bf786faf9a411365cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:19 UTC) from {username='slave'} at 127.0.0.1:37542
I20260928 12:26:19.382741 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3ce093d7ec4bf786faf9a411365cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:19 UTC) from {username='slave'} at 127.0.0.1:46370
I20260928 12:26:19.382928 12355 tablet_replica.cc:330] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: stopping tablet replica
I20260928 12:26:19.382934 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11927
I20260928 12:26:19.383024 12355 raft_consensus.cc:2229] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:19.383533 12355 raft_consensus.cc:2258] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:19.383692 12353 tablet_replica.cc:330] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: stopping tablet replica
I20260928 12:26:19.383783 12353 raft_consensus.cc:2229] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:19.383870 12353 raft_consensus.cc:2258] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:19.384284 12353 ts_tablet_manager.cc:1905] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:19.385046 12355 ts_tablet_manager.cc:1905] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:19.386035 12353 ts_tablet_manager.cc:1918] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:19.386112 12353 log.cc:1196] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/2c3ce093d7ec4bf786faf9a411365cf6
I20260928 12:26:19.386366 12353 ts_tablet_manager.cc:1939] T 2c3ce093d7ec4bf786faf9a411365cf6 P e80c12b0645d4bd988d857ded26c5b86: Deleting consensus metadata
I20260928 12:26:19.386745 12355 ts_tablet_manager.cc:1918] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:19.386813 12355 log.cc:1196] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/2c3ce093d7ec4bf786faf9a411365cf6
I20260928 12:26:19.386823 11852 catalog_manager.cc:4841] TS e80c12b0645d4bd988d857ded26c5b86 (127.7.19.67:35429): tablet 2c3ce093d7ec4bf786faf9a411365cf6 (table test-table [id=12bb594078a648ac8834be329f21e3d0]) successfully deleted
I20260928 12:26:19.387079 12355 ts_tablet_manager.cc:1939] T 2c3ce093d7ec4bf786faf9a411365cf6 P ee9239ad9ddf4701af431ec230a74b9d: Deleting consensus metadata
I20260928 12:26:19.387461 11852 catalog_manager.cc:4841] TS ee9239ad9ddf4701af431ec230a74b9d (127.7.19.66:38169): tablet 2c3ce093d7ec4bf786faf9a411365cf6 (table test-table [id=12bb594078a648ac8834be329f21e3d0]) successfully deleted
W20260928 12:26:19.388867 11852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:35077 (error 108)
I20260928 12:26:19.389082 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12058
I20260928 12:26:19.389809 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.389236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:35077 (local address 127.0.0.1:58012)
0928 12:26:19.389731 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.389771 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
W20260928 12:26:19.390622 11852 catalog_manager.cc:4584] TS 8bdcdab0cbd6410fb72f3b00d7691aaa (127.7.19.65:35077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c3ce093d7ec4bf786faf9a411365cf6: Network error: Client connection negotiation failed: client connection to 127.7.19.65:35077: connect: Connection refused (error 111)
I20260928 12:26:19.393683 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12189
I20260928 12:26:19.398638 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 11836
2026-09-28T12:26:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (578 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 12:26:19.418696 7245 mini_oidc.cc:152] Starting JWKS server
I20260928 12:26:19.419750 7245 webserver.cc:460] Webserver started at https://127.0.0.1:38713/ using document root <none> and password file <none>
I20260928 12:26:19.419869 7245 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:26:19.420054 7245 webserver.cc:460] Webserver started at http://127.0.0.1:33805/ using document root <none> and password file <none>
2026-09-28T12:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:19Z Disabled control of system clock
I20260928 12:26:19.427438 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39549
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:39549
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38713/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/testchainca.cert with env {}
W20260928 12:26:19.505604 12362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:19.505812 12362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:19.505834 12362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:19.507118 12362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:19.507162 12362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:19.507176 12362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:19.507189 12362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:19.507201 12362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:26:19.507258 12362 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:26:19.507275 12362 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38713/jwks.json
I20260928 12:26:19.508790 12362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:39549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38713/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:19.509037 12362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:19.509262 12362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:19.514076 12362 server_base.cc:1034] running on GCE node
W20260928 12:26:19.514053 12367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.514034 12370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.514046 12368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.514472 12362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:19.514755 12362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:19.515921 12362 hybrid_clock.cc:648] HybridClock initialized: now 1790598379515916 us; error 36 us; skew 500 ppm
I20260928 12:26:19.517200 12362 webserver.cc:460] Webserver started at http://127.7.19.126:34091/ using document root <none> and password file <none>
I20260928 12:26:19.517479 12362 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:19.517553 12362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:19.517697 12362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:19.518708 12362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "ca54cd280cc34670ada153bcc960ab46"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "5b26e46fcb0af1d51f478b0f085bc1bb"
server_key_iv: "253ae43fc7204e36866a28842dec5a9e"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.519068 12362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "ca54cd280cc34670ada153bcc960ab46"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "5b26e46fcb0af1d51f478b0f085bc1bb"
server_key_iv: "253ae43fc7204e36866a28842dec5a9e"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.520387 12362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:19.521193 12376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.521400 12362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:19.521497 12362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "ca54cd280cc34670ada153bcc960ab46"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "5b26e46fcb0af1d51f478b0f085bc1bb"
server_key_iv: "253ae43fc7204e36866a28842dec5a9e"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.521588 12362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:19.540231 12362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:19.540598 12362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:19.540800 12362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:19.544652 12362 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:39549
I20260928 12:26:19.544730 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:39549 every 8 connection(s)
I20260928 12:26:19.545246 12362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:19.546219 12429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:19.548539 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46: Bootstrap starting.
I20260928 12:26:19.549153 12429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:19.549454 12429 log.cc:824] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:19.550099 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46: No bootstrap required, opened a new log
I20260928 12:26:19.551235 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12362
I20260928 12:26:19.551402 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:19.551594 7245 external_mini_cluster.cc:1428] Setting key 710cce45e120dbff356da1252271eb91
I20260928 12:26:19.551994 12429 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca54cd280cc34670ada153bcc960ab46" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39549 } }
I20260928 12:26:19.552178 12429 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:19.552223 12429 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca54cd280cc34670ada153bcc960ab46, State: Initialized, Role: FOLLOWER
I20260928 12:26:19.552358 12429 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca54cd280cc34670ada153bcc960ab46" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39549 } }
I20260928 12:26:19.552452 12429 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:19.552496 12429 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:19.552547 12429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:19.553259 12429 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca54cd280cc34670ada153bcc960ab46" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39549 } }
I20260928 12:26:19.553385 12429 leader_election.cc:302] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca54cd280cc34670ada153bcc960ab46; no voters:
I20260928 12:26:19.553555 12429 leader_election.cc:288] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:19.553630 12434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:19.553793 12434 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [term 1 LEADER]: Becoming Leader. State: Replica: ca54cd280cc34670ada153bcc960ab46, State: Running, Role: LEADER
I20260928 12:26:19.553813 12429 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:19.553879 12434 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca54cd280cc34670ada153bcc960ab46" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39549 } }
I20260928 12:26:19.554399 12435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca54cd280cc34670ada153bcc960ab46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca54cd280cc34670ada153bcc960ab46" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39549 } } }
I20260928 12:26:19.554504 12435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:19.554652 12436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca54cd280cc34670ada153bcc960ab46. Latest consensus state: current_term: 1 leader_uuid: "ca54cd280cc34670ada153bcc960ab46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca54cd280cc34670ada153bcc960ab46" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39549 } } }
I20260928 12:26:19.554723 12436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:19.554836 12443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:19.555320 12443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:19.558075 12443 catalog_manager.cc:1347] Generated new cluster ID: 2bd9b9381cba4bf897f4c8e6abe38914
I20260928 12:26:19.558128 12443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:19.558162 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.552174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.7.19.126:39549)
0928 12:26:19.552404 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.552414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.552436 (+ 22us) server_negotiation.cc:407] Connection header received
0928 12:26:19.552565 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.552572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.552661 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.552822 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.553036 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.554759 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.555939 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.556591 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.556732 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:19.557467 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.557470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.557470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.557674 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.557715 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.557729 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.557809 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.557884 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.557962 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260928 12:26:19.559021 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:26:19.571956 12443 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:19.572557 12443 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:19.574924 12443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca54cd280cc34670ada153bcc960ab46: Generated new TSK 0
I20260928 12:26:19.575107 12443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:19.599012 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.582799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.7.19.126:39549)
0928 12:26:19.582862 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.582864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.582966 (+ 102us) server_negotiation.cc:407] Connection header received
0928 12:26:19.583052 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.583053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.583065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.583146 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:26:19.583365 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.584127 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.585014 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.585318 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.585388 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:19.585599 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:26:19.598765 (+ 13166us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:26:19.598931 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 12:26:19.599092 12433 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
E20260928 12:26:19.599318 12459 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 12:26:19.599393 12459 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 12:26:19.599444 12458 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:19.582707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39549 (local address 127.0.0.1:45050)
0928 12:26:19.582898 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.582908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.582962 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.583213 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.583214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.583216 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:26:19.583279 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:19.583281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.584222 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.584223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:19.584929 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:19.584932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.585400 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.585401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:19.585452 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:19.585472 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:26:19.598911 (+ 13439us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 12:26:19.599068 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.19.126:39549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.19.126:39549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38713/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 12:26:19.599926 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12362
2026-09-28T12:26:19Z chronyd exiting
I20260928 12:26:19.635893 7245 test_util.cc:175] -----------------------------------------------
I20260928 12:26:19.636003 7245 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790598357699237-7245-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 12:26:19.636824 7245 mini_oidc.cc:152] Starting JWKS server
I20260928 12:26:19.637535 7245 webserver.cc:460] Webserver started at https://127.0.0.1:35129/ using document root <none> and password file <none>
I20260928 12:26:19.637603 7245 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:26:19.637766 7245 webserver.cc:460] Webserver started at http://127.0.0.1:43341/ using document root <none> and password file <none>
2026-09-28T12:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:19Z Disabled control of system clock
I20260928 12:26:19.645517 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39729
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:39729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35129/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/testchainca.cert with env {}
W20260928 12:26:19.727624 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:19.727850 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:19.727873 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:19.729101 12465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:19.729146 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:19.729159 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:19.729172 12465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:19.729184 12465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:26:19.729197 12465 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:26:19.729255 12465 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35129/jwks.json
I20260928 12:26:19.730769 12465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:39729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35129/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:19.731130 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:19.731390 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:19.736461 12473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.736505 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.736537 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.736828 12465 server_base.cc:1034] running on GCE node
I20260928 12:26:19.737144 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:19.737470 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:19.738667 12465 hybrid_clock.cc:648] HybridClock initialized: now 1790598379738642 us; error 44 us; skew 500 ppm
I20260928 12:26:19.740056 12465 webserver.cc:460] Webserver started at http://127.7.19.126:39069/ using document root <none> and password file <none>
I20260928 12:26:19.740358 12465 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:19.740447 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:19.740612 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:19.741816 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "50ac277438984e1a83625c525e61c6cc"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "6ad2aabdb62e4a0b990fa4206e2ba3b8"
server_key_iv: "333e72929b590388c12b8a0a900dc4a6"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.742219 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "50ac277438984e1a83625c525e61c6cc"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "6ad2aabdb62e4a0b990fa4206e2ba3b8"
server_key_iv: "333e72929b590388c12b8a0a900dc4a6"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.743677 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:19.744482 12479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.744737 12465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:19.744812 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "50ac277438984e1a83625c525e61c6cc"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "6ad2aabdb62e4a0b990fa4206e2ba3b8"
server_key_iv: "333e72929b590388c12b8a0a900dc4a6"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.744935 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:19.759894 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:19.760200 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:19.760345 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:19.764593 12465 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:39729
I20260928 12:26:19.764665 12531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:39729 every 8 connection(s)
I20260928 12:26:19.765002 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:19.765962 12532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:19.768174 12532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc: Bootstrap starting.
I20260928 12:26:19.768802 12532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:19.769078 12532 log.cc:824] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:19.769641 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12465
I20260928 12:26:19.769743 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:19.769853 12532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc: No bootstrap required, opened a new log
I20260928 12:26:19.769927 7245 external_mini_cluster.cc:1428] Setting key 40f880979c046021b3258e0a44018992
I20260928 12:26:19.771852 12532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ac277438984e1a83625c525e61c6cc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39729 } }
I20260928 12:26:19.771978 12532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:19.772001 12532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ac277438984e1a83625c525e61c6cc, State: Initialized, Role: FOLLOWER
I20260928 12:26:19.772125 12532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ac277438984e1a83625c525e61c6cc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39729 } }
I20260928 12:26:19.772197 12532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:19.772218 12532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:19.772285 12532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:19.772891 12532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ac277438984e1a83625c525e61c6cc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39729 } }
I20260928 12:26:19.772996 12532 leader_election.cc:302] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ac277438984e1a83625c525e61c6cc; no voters:
I20260928 12:26:19.773161 12532 leader_election.cc:288] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:19.773219 12537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:19.773360 12537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [term 1 LEADER]: Becoming Leader. State: Replica: 50ac277438984e1a83625c525e61c6cc, State: Running, Role: LEADER
I20260928 12:26:19.773409 12532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:19.773447 12537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ac277438984e1a83625c525e61c6cc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39729 } }
I20260928 12:26:19.773715 12537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ac277438984e1a83625c525e61c6cc. Latest consensus state: current_term: 1 leader_uuid: "50ac277438984e1a83625c525e61c6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ac277438984e1a83625c525e61c6cc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39729 } } }
I20260928 12:26:19.773802 12537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:19.773722 12539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ac277438984e1a83625c525e61c6cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ac277438984e1a83625c525e61c6cc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39729 } } }
I20260928 12:26:19.773960 12539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:19.774080 12543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:19.774544 12543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:19.776350 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.770422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55012 (local address 127.7.19.126:39729)
0928 12:26:19.770683 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.770697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.770722 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:26:19.771171 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.771177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.771233 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.771383 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:19.771624 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.773021 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.774231 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.774812 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.774953 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:19.775605 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:19.775609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:19.775609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:19.775800 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:19.775841 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:19.775856 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:19.775918 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:19.776057 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:26:19.776141 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260928 12:26:19.777372 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:26:19.777408 12543 catalog_manager.cc:1347] Generated new cluster ID: 5bab60c28eaa469aa0ec0203b2677606
I20260928 12:26:19.777444 12543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:19.790418 12543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:19.791211 12543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:19.795977 12543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ac277438984e1a83625c525e61c6cc: Generated new TSK 0
I20260928 12:26:19.796177 12543 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 12:26:19.817860 12562 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 12:26:19.817960 12562 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 12:26:19.818325 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:19.803446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.7.19.126:39729)
0928 12:26:19.803520 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:19.803523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:19.803741 (+ 218us) server_negotiation.cc:407] Connection header received
0928 12:26:19.803776 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:19.803777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:19.803789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:19.803848 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:26:19.804004 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.804510 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.805647 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.805854 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.805909 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:19.806090 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:26:19.818123 (+ 12033us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:26:19.818272 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 12:26:19.818399 12536 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 12:26:19.818522 12561 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:19.803394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39729 (local address 127.0.0.1:55014)
0928 12:26:19.803602 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:19.803614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:19.803687 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:19.803871 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:19.803873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:19.803876 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:26:19.803934 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:19.803937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.804586 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.804586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:19.805558 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:19.805560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:19.805929 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:19.805930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:19.805991 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:19.805997 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:26:19.818268 (+ 12271us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 12:26:19.818416 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.19.126:39729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":144,"threads_started":1}
I20260928 12:26:19.818826 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12465
2026-09-28T12:26:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 12:26:19.857317 7245 mini_oidc.cc:152] Starting JWKS server
I20260928 12:26:19.858026 7245 webserver.cc:460] Webserver started at https://127.0.0.1:41305/ using document root <none> and password file <none>
I20260928 12:26:19.858069 7245 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 12:26:19.858230 7245 webserver.cc:460] Webserver started at http://127.0.0.1:36001/ using document root <none> and password file <none>
2026-09-28T12:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:19Z Disabled control of system clock
I20260928 12:26:19.866143 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37397
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37397
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41305/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:26:19.947867 12568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:19.948050 12568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:19.948071 12568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:19.949149 12568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:19.949182 12568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:19.949195 12568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:19.949208 12568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:19.949219 12568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:26:19.949232 12568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:26:19.949249 12568 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41305/jwks.json
I20260928 12:26:19.950592 12568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41305/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:19.950773 12568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:19.950990 12568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:19.955678 12576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.955761 12573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:19.955680 12574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:19.955888 12568 server_base.cc:1034] running on GCE node
I20260928 12:26:19.956137 12568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:19.956377 12568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:19.957525 12568 hybrid_clock.cc:648] HybridClock initialized: now 1790598379957499 us; error 39 us; skew 500 ppm
I20260928 12:26:19.958715 12568 webserver.cc:460] Webserver started at http://127.7.19.126:35577/ using document root <none> and password file <none>
I20260928 12:26:19.958977 12568 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:19.959026 12568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:19.959156 12568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:19.960273 12568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "8cf0fc1611024e699080dd3ec615dd91"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "8b16797077718f22afbb5ebdbe99b170"
server_key_iv: "be857407c46eaafe03dff9d47444c734"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.960644 12568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "8cf0fc1611024e699080dd3ec615dd91"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "8b16797077718f22afbb5ebdbe99b170"
server_key_iv: "be857407c46eaafe03dff9d47444c734"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.961995 12568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:19.962667 12582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:19.962839 12568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:19.962919 12568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "8cf0fc1611024e699080dd3ec615dd91"
format_stamp: "Formatted at 2026-09-28 12:26:19 on dist-test-slave-cb7x"
server_key: "8b16797077718f22afbb5ebdbe99b170"
server_key_iv: "be857407c46eaafe03dff9d47444c734"
server_key_version: "encryptionkey@0"
I20260928 12:26:19.962998 12568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:19.991065 12568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:19.991309 12568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:19.991412 12568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:19.994979 12568 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:37397
I20260928 12:26:19.995029 12634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:37397 every 8 connection(s)
I20260928 12:26:19.995340 12568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:19.996052 12635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:19.998798 12635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91: Bootstrap starting.
I20260928 12:26:19.999392 12635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:19.999682 12635 log.cc:824] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:20.000412 12635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91: No bootstrap required, opened a new log
I20260928 12:26:20.000634 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12568
I20260928 12:26:20.000768 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:20.000981 7245 external_mini_cluster.cc:1428] Setting key a13c535a5d5ba5088591749794b39b5a
I20260928 12:26:20.002424 12635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf0fc1611024e699080dd3ec615dd91" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37397 } }
I20260928 12:26:20.002578 12635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:20.002624 12635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf0fc1611024e699080dd3ec615dd91, State: Initialized, Role: FOLLOWER
I20260928 12:26:20.002758 12635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf0fc1611024e699080dd3ec615dd91" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37397 } }
I20260928 12:26:20.002857 12635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:20.002899 12635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:20.002952 12635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:20.003468 12635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf0fc1611024e699080dd3ec615dd91" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37397 } }
I20260928 12:26:20.003583 12635 leader_election.cc:302] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cf0fc1611024e699080dd3ec615dd91; no voters:
I20260928 12:26:20.003762 12635 leader_election.cc:288] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:20.003814 12640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:20.003937 12640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [term 1 LEADER]: Becoming Leader. State: Replica: 8cf0fc1611024e699080dd3ec615dd91, State: Running, Role: LEADER
I20260928 12:26:20.004016 12640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf0fc1611024e699080dd3ec615dd91" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37397 } }
I20260928 12:26:20.004019 12635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:20.004281 12640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf0fc1611024e699080dd3ec615dd91. Latest consensus state: current_term: 1 leader_uuid: "8cf0fc1611024e699080dd3ec615dd91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf0fc1611024e699080dd3ec615dd91" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37397 } } }
I20260928 12:26:20.004335 12640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:20.004410 12642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cf0fc1611024e699080dd3ec615dd91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf0fc1611024e699080dd3ec615dd91" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37397 } } }
I20260928 12:26:20.004519 12642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:20.004824 12644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:20.005339 12644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:20.006966 12644 catalog_manager.cc:1347] Generated new cluster ID: af731038cb074b51963c5c1aa2d6b527
I20260928 12:26:20.007017 12644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:20.008142 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:20.001488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.7.19.126:37397)
0928 12:26:20.001702 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:20.001709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:20.001733 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:26:20.002404 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:20.002410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:20.002482 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:20.002624 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:20.002874 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.004526 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.005760 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.006286 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.006416 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:20.007368 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:20.007372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:20.007372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:20.007570 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:20.007612 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:20.007625 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:20.007696 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:20.007764 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:26:20.007941 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260928 12:26:20.009058 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:26:20.016531 12644 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:20.017076 12644 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:20.022115 12644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cf0fc1611024e699080dd3ec615dd91: Generated new TSK 0
I20260928 12:26:20.022295 12644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:20.030351 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:20.026669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.7.19.126:37397)
0928 12:26:20.026709 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:20.026711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:20.026775 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:26:20.026834 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:20.026835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:20.026847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:20.026938 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:26:20.027035 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.027780 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.028623 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.028968 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.029040 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:20.029211 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:26:20.030143 (+ 932us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41305/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:26:20.030275 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41305/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:26:20.030407 12639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41305/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 12:26:20.032312 12664 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:20.026590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37397 (local address 127.0.0.1:37408)
0928 12:26:20.026706 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:20.026716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:20.026778 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:20.026929 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:20.026929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:20.026931 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:26:20.026976 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:20.026978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.027906 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.027908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:20.028538 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:20.028540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.029052 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.029053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:20.029101 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:20.029106 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:26:20.030252 (+ 1146us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41305/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 12:26:20.032210 (+ 1958us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.19.126:37397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41305/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":1744,"thread_start_us":75,"threads_started":1}
I20260928 12:26:20.032579 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12568
E20260928 12:26:20.041376 12665 webserver.cc:533] Webserver: sslize failed:
E20260928 12:26:20.041451 12665 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T12:26:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T12:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:20Z Disabled control of system clock
I20260928 12:26:20.078486 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38079
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:45091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:38079
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/testchainca.cert with env {}
W20260928 12:26:20.158716 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:20.158923 12669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:20.158946 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:20.160126 12669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:20.160173 12669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:20.160187 12669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:20.160199 12669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:20.160211 12669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 12:26:20.160274 12669 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 12:26:20.160290 12669 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 12:26:20.161839 12669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:38079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:20.162096 12669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:20.162315 12669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:20.167471 12677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:20.167515 12674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:20.167515 12675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:20.167589 12669 server_base.cc:1034] running on GCE node
I20260928 12:26:20.167979 12669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:20.168213 12669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:20.169360 12669 hybrid_clock.cc:648] HybridClock initialized: now 1790598380169344 us; error 27 us; skew 500 ppm
I20260928 12:26:20.170584 12669 webserver.cc:460] Webserver started at http://127.7.19.126:41941/ using document root <none> and password file <none>
I20260928 12:26:20.170823 12669 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:20.170892 12669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:20.171031 12669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:20.172154 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "a73c7cb599784675956c4c6d4a9a7ce6"
format_stamp: "Formatted at 2026-09-28 12:26:20 on dist-test-slave-cb7x"
server_key: "530ef3ca47fc94ec2234b2158a025545"
server_key_iv: "b6d21ff63619b526736f0225e114e821"
server_key_version: "encryptionkey@0"
I20260928 12:26:20.172514 12669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "a73c7cb599784675956c4c6d4a9a7ce6"
format_stamp: "Formatted at 2026-09-28 12:26:20 on dist-test-slave-cb7x"
server_key: "530ef3ca47fc94ec2234b2158a025545"
server_key_iv: "b6d21ff63619b526736f0225e114e821"
server_key_version: "encryptionkey@0"
I20260928 12:26:20.173856 12669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:20.174714 12683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:20.174923 12669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:20.174995 12669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "a73c7cb599784675956c4c6d4a9a7ce6"
format_stamp: "Formatted at 2026-09-28 12:26:20 on dist-test-slave-cb7x"
server_key: "530ef3ca47fc94ec2234b2158a025545"
server_key_iv: "b6d21ff63619b526736f0225e114e821"
server_key_version: "encryptionkey@0"
I20260928 12:26:20.175084 12669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:20.191972 12669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:20.192282 12669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:20.192420 12669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:20.196362 12669 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:38079
I20260928 12:26:20.196440 12735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:38079 every 8 connection(s)
I20260928 12:26:20.196769 12669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:20.197755 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:20.199975 12736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6: Bootstrap starting.
I20260928 12:26:20.200546 12736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:20.200873 12736 log.cc:824] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:20.201557 12736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6: No bootstrap required, opened a new log
I20260928 12:26:20.202898 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12669
I20260928 12:26:20.203006 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:20.203190 7245 external_mini_cluster.cc:1428] Setting key 7924d9e06dd6bec6081e983fa0287f6f
I20260928 12:26:20.203447 12736 raft_consensus.cc:348] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38079 } }
I20260928 12:26:20.203667 12736 raft_consensus.cc:374] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:20.203717 12736 raft_consensus.cc:729] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73c7cb599784675956c4c6d4a9a7ce6, State: Initialized, Role: FOLLOWER
I20260928 12:26:20.203866 12736 consensus_queue.cc:260] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38079 } }
I20260928 12:26:20.203969 12736 raft_consensus.cc:388] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:20.204010 12736 raft_consensus.cc:482] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:20.204059 12736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:20.204792 12736 raft_consensus.cc:504] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38079 } }
I20260928 12:26:20.204908 12736 leader_election.cc:302] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a73c7cb599784675956c4c6d4a9a7ce6; no voters:
I20260928 12:26:20.205066 12736 leader_election.cc:288] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:20.205204 12741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:20.205315 12736 sys_catalog.cc:564] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:20.205333 12741 raft_consensus.cc:686] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [term 1 LEADER]: Becoming Leader. State: Replica: a73c7cb599784675956c4c6d4a9a7ce6, State: Running, Role: LEADER
I20260928 12:26:20.205405 12741 consensus_queue.cc:237] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38079 } }
I20260928 12:26:20.205688 12741 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73c7cb599784675956c4c6d4a9a7ce6. Latest consensus state: current_term: 1 leader_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38079 } } }
I20260928 12:26:20.205745 12741 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:20.205703 12742 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7cb599784675956c4c6d4a9a7ce6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38079 } } }
I20260928 12:26:20.205919 12742 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:20.206682 12747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:20.207327 12747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:20.208966 12747 catalog_manager.cc:1347] Generated new cluster ID: a4fd9efe96ce402db0b718b11a5d0370
I20260928 12:26:20.209036 12747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:20.209199 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:20.203680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.7.19.126:38079)
0928 12:26:20.203890 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:20.203896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:20.203919 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:26:20.204051 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:20.204058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:20.204146 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:20.204264 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:20.204524 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.205805 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.206950 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.207501 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.207638 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:20.208334 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:20.208337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:20.208338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:20.208540 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:20.208580 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:20.208594 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:20.208662 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:20.208792 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:26:20.208889 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260928 12:26:20.210022 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:26:20.240306 12747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:20.240927 12747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:20.250061 12747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a73c7cb599784675956c4c6d4a9a7ce6: Generated new TSK 0
I20260928 12:26:20.250279 12747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:20.382407 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:20.262974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.7.19.126:38079)
0928 12:26:20.263042 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:20.263044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:20.263132 (+ 88us) server_negotiation.cc:407] Connection header received
0928 12:26:20.263176 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:20.263177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:20.263190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:20.263275 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0928 12:26:20.263396 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.264185 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.265010 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.265327 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.265402 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:20.265565 (+ 163us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 12:26:20.382179 (+116614us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:26:20.382344 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42546: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 12:26:20.382471 12740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42546: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 12:26:20.382659 12765 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:20.262887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38079 (local address 127.0.0.1:42546)
0928 12:26:20.263064 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:20.263074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:20.263126 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:20.263278 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:20.263279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:20.263281 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 12:26:20.263329 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:20.263330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.264278 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.264279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:20.264915 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:20.264917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:20.265408 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:20.265409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:20.265457 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:20.265463 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 12:26:20.382362 (+116899us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 12:26:20.382527 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.19.126:38079: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260928 12:26:20.383014 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12669
2026-09-28T12:26:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (330 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:20 dist-test-slave-cb7x krb5kdc[12777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:20 dist-test-slave-cb7x krb5kdc[12777](info): set up 2 sockets
Sep 28 12:26:20 dist-test-slave-cb7x krb5kdc[12777](info): commencing operation
krb5kdc: starting...
W20260928 12:26:21.588876 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:21 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:21.707907 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34743
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:21.788172 12790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:21.788434 12790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:21.788473 12790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:21.789681 12790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:21.789750 12790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:21.789778 12790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:21.789799 12790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:21.789830 12790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:21.791313 12790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:21.791587 12790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:21.791816 12790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:21.796588 12798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:21.796592 12795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:21.796635 12796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:21.796859 12790 server_base.cc:1034] running on GCE node
I20260928 12:26:21.797240 12790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:21.797545 12790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:21.798728 12790 hybrid_clock.cc:648] HybridClock initialized: now 1790598381798706 us; error 37 us; skew 500 ppm
Sep 28 12:26:21 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598381, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:21.801380 12790 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:21.801822 12790 webserver.cc:460] Webserver started at http://127.7.19.126:43159/ using document root <none> and password file <none>
I20260928 12:26:21.802162 12790 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:21.802230 12790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:21.802378 12790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:21.803478 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "34eca51f34664d53b5e179c313fdd815"
format_stamp: "Formatted at 2026-09-28 12:26:21 on dist-test-slave-cb7x"
server_key: "a6b98a0cd3982275bd2e3ac998fed03a"
server_key_iv: "0a3029dc4f84838a252e2a9229e73866"
server_key_version: "encryptionkey@0"
I20260928 12:26:21.804235 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "34eca51f34664d53b5e179c313fdd815"
format_stamp: "Formatted at 2026-09-28 12:26:21 on dist-test-slave-cb7x"
server_key: "a6b98a0cd3982275bd2e3ac998fed03a"
server_key_iv: "0a3029dc4f84838a252e2a9229e73866"
server_key_version: "encryptionkey@0"
I20260928 12:26:21.805670 12790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 12:26:21.806529 12805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:21.806707 12790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:21.806826 12790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "34eca51f34664d53b5e179c313fdd815"
format_stamp: "Formatted at 2026-09-28 12:26:21 on dist-test-slave-cb7x"
server_key: "a6b98a0cd3982275bd2e3ac998fed03a"
server_key_iv: "0a3029dc4f84838a252e2a9229e73866"
server_key_version: "encryptionkey@0"
I20260928 12:26:21.806921 12790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:21.821041 12790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:21.822992 12790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:21.823153 12790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:21.827203 12790 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:34743
I20260928 12:26:21.827239 12857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:34743 every 8 connection(s)
I20260928 12:26:21.827633 12790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:21.828504 12858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:21.830754 12858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815: Bootstrap starting.
I20260928 12:26:21.831310 12858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:21.831600 12858 log.cc:824] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:21.832317 12858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815: No bootstrap required, opened a new log
I20260928 12:26:21.832731 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12790
I20260928 12:26:21.832854 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:21.833061 7245 external_mini_cluster.cc:1428] Setting key 8c93a026f9b2085f970410e3b2d4fa10
I20260928 12:26:21.834858 12858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34eca51f34664d53b5e179c313fdd815" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34743 } }
I20260928 12:26:21.835039 12858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:21.835085 12858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34eca51f34664d53b5e179c313fdd815, State: Initialized, Role: FOLLOWER
I20260928 12:26:21.835227 12858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34eca51f34664d53b5e179c313fdd815" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34743 } }
I20260928 12:26:21.835317 12858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:21.835352 12858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:21.835404 12858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:26:21 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:21.843348 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:21.833583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.7.19.126:34743)
0928 12:26:21.833784 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:21.833791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:21.833815 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:26:21.834797 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:21.834805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:21.834854 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:21.834978 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:21.835793 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:21.836756 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:21.837485 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:21.837901 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:21.837998 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:21.840353 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:21.840357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:21.840358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:21.840371 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:21.841981 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:21.842353 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:21.842356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:21.842356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:21.842438 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:21.842603 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:21.842604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:21.842604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:21.842825 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:21.842923 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:21.843018 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 12:26:21.843149 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260928 12:26:21.868083 12858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34eca51f34664d53b5e179c313fdd815" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34743 } }
I20260928 12:26:21.868449 12858 leader_election.cc:302] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34eca51f34664d53b5e179c313fdd815; no voters:
I20260928 12:26:21.868754 12858 leader_election.cc:288] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:21.868934 12863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:21.869107 12858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:21.869125 12863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [term 1 LEADER]: Becoming Leader. State: Replica: 34eca51f34664d53b5e179c313fdd815, State: Running, Role: LEADER
I20260928 12:26:21.869218 12863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34eca51f34664d53b5e179c313fdd815" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34743 } }
I20260928 12:26:21.869705 12864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34eca51f34664d53b5e179c313fdd815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34eca51f34664d53b5e179c313fdd815" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34743 } } }
I20260928 12:26:21.869815 12864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:21.870069 12865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34eca51f34664d53b5e179c313fdd815. Latest consensus state: current_term: 1 leader_uuid: "34eca51f34664d53b5e179c313fdd815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34eca51f34664d53b5e179c313fdd815" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34743 } } }
I20260928 12:26:21.870187 12865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:21.870613 12872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:21.871156 12872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:21.872956 12872 catalog_manager.cc:1347] Generated new cluster ID: 75e0570ee9b249aa87d397be3db434ad
I20260928 12:26:21.873019 12872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:21.889050 12872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:21.889660 12872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:21.897336 12872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34eca51f34664d53b5e179c313fdd815: Generated new TSK 0
I20260928 12:26:21.897553 12872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
W20260928 12:26:22.788846 7245 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.7.19.65: real 0.888s user 0.001s sys 0.002s
I20260928 12:26:22.789206 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34743
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:22.872030 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:22.872226 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:22.872248 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:22.873358 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:22.873399 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:22.873423 12884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:22.874926 12884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:22.875113 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:22.875273 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:22.880602 12890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:22.880604 12892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:22.880661 12884 server_base.cc:1034] running on GCE node
W20260928 12:26:22.880604 12889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:22.881131 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:22.881377 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:22.882529 12884 hybrid_clock.cc:648] HybridClock initialized: now 1790598382882516 us; error 30 us; skew 500 ppm
Sep 28 12:26:22 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:22.885130 12884 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:22.885560 12884 webserver.cc:460] Webserver started at http://127.7.19.65:46857/ using document root <none> and password file <none>
I20260928 12:26:22.885798 12884 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:22.885866 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:22.886021 12884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:22.887063 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "589098bd04f741a28ef2bb24a5f4723b"
format_stamp: "Formatted at 2026-09-28 12:26:22 on dist-test-slave-cb7x"
server_key: "b970d53a0462e0ff10a0eb966d12a340"
server_key_iv: "44727bf653452d718c258c07a502deb0"
server_key_version: "encryptionkey@0"
I20260928 12:26:22.887426 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "589098bd04f741a28ef2bb24a5f4723b"
format_stamp: "Formatted at 2026-09-28 12:26:22 on dist-test-slave-cb7x"
server_key: "b970d53a0462e0ff10a0eb966d12a340"
server_key_iv: "44727bf653452d718c258c07a502deb0"
server_key_version: "encryptionkey@0"
I20260928 12:26:22.888789 12884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:22.889559 12899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:22.889737 12884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:22.889835 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "589098bd04f741a28ef2bb24a5f4723b"
format_stamp: "Formatted at 2026-09-28 12:26:22 on dist-test-slave-cb7x"
server_key: "b970d53a0462e0ff10a0eb966d12a340"
server_key_iv: "44727bf653452d718c258c07a502deb0"
server_key_version: "encryptionkey@0"
I20260928 12:26:22.889921 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:22.919697 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:22.921710 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:22.921873 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:22.922102 12884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:22.922397 12884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:22.922472 12884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:22.922513 12884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:22.922540 12884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:22.928107 12884 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:42413
I20260928 12:26:22.928153 13012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:42413 every 8 connection(s)
I20260928 12:26:22.928457 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:22.933945 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 12884
I20260928 12:26:22.934073 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:22.934281 7245 external_mini_cluster.cc:1428] Setting key 935aff102e48cad53a8ac1bc4738896a
Sep 28 12:26:22 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598382, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:22.937980 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:22.929166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:34329 (local address 127.7.19.126:34743)
0928 12:26:22.929288 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:22.929290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:22.929424 (+ 134us) server_negotiation.cc:407] Connection header received
0928 12:26:22.930345 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:22.930345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:22.930356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:22.930421 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:22.931020 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:22.931515 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:22.932534 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:22.932762 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:22.932836 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:22.935083 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:22.935086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:22.935086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:22.935093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:22.936459 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:22.936960 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:22.936961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:22.936961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:22.937029 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:22.937327 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:22.937328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:22.937328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:22.937449 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:22.937510 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:22.937832 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 12:26:22.937901 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260928 12:26:22.938271 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:22.929140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34743 (local address 127.7.19.65:34329)
0928 12:26:22.929315 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:22.929356 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:22.930253 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:22.930501 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:22.930505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:22.930767 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:22.930948 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:22.930952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:22.931607 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:22.931608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:22.932435 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:22.932438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:22.932842 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:22.932843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:22.932960 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:22.933403 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:22.933411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:22.934984 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:22.936555 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:22.936561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:22.936563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:22.936861 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:22.937107 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:22.937108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:22.937108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:22.937269 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:22.937521 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:22.937527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:22.937697 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 12:26:22.937811 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 12:26:22.937985 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260928 12:26:22.938838 13013 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34743
I20260928 12:26:22.938931 13013 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:22.939122 13013 heartbeater.cc:511] Master 127.7.19.126:34743 requested a full tablet report, sending...
I20260928 12:26:22.939678 12822 ts_manager.cc:194] Registered new tserver with Master: 589098bd04f741a28ef2bb24a5f4723b (127.7.19.65:42413)
I20260928 12:26:22.940308 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:34329
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:22.959447 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34743
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:23.039613 13019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:23.039805 13019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:23.039829 13019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:23.041000 13019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:23.041039 13019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:23.041065 13019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:23.042443 13019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:23.042618 13019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:23.042783 13019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:23.047610 13027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:23.047659 13025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:23.047753 13019 server_base.cc:1034] running on GCE node
W20260928 12:26:23.047819 13024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:23.048058 13019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:23.048288 13019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:23.049444 13019 hybrid_clock.cc:648] HybridClock initialized: now 1790598383049439 us; error 35 us; skew 500 ppm
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.051862 13019 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:23.052301 13019 webserver.cc:460] Webserver started at http://127.7.19.66:35649/ using document root <none> and password file <none>
I20260928 12:26:23.052496 13019 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:23.052538 13019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:23.052623 13019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:23.053746 13019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "502ff2f511a9407fbc47ef1180ba7cf6"
format_stamp: "Formatted at 2026-09-28 12:26:23 on dist-test-slave-cb7x"
server_key: "ef98f1285adac0b922cca4349c373695"
server_key_iv: "3bd1ecad71c9c51184306c8eb9bc72a8"
server_key_version: "encryptionkey@0"
I20260928 12:26:23.054051 13019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "502ff2f511a9407fbc47ef1180ba7cf6"
format_stamp: "Formatted at 2026-09-28 12:26:23 on dist-test-slave-cb7x"
server_key: "ef98f1285adac0b922cca4349c373695"
server_key_iv: "3bd1ecad71c9c51184306c8eb9bc72a8"
server_key_version: "encryptionkey@0"
I20260928 12:26:23.055258 13019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:23.055863 13034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:23.056011 13019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:23.056092 13019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "502ff2f511a9407fbc47ef1180ba7cf6"
format_stamp: "Formatted at 2026-09-28 12:26:23 on dist-test-slave-cb7x"
server_key: "ef98f1285adac0b922cca4349c373695"
server_key_iv: "3bd1ecad71c9c51184306c8eb9bc72a8"
server_key_version: "encryptionkey@0"
I20260928 12:26:23.056145 13019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:23.066423 13019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:23.068544 13019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:23.068648 13019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:23.068895 13019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:23.069198 13019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:23.069231 13019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:23.069253 13019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:23.069267 13019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:23.074843 13019 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:45983
I20260928 12:26:23.074920 13147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:45983 every 8 connection(s)
I20260928 12:26:23.075205 13019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:23.083956 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13019
I20260928 12:26:23.084052 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:23.084230 7245 external_mini_cluster.cc:1428] Setting key c5b2db0270f0ea9308e68e1eb61d1cbf
I20260928 12:26:23.091003 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.075880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34743 (local address 127.7.19.66:52457)
0928 12:26:23.076043 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:23.076084 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:23.077112 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:23.077416 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:23.077419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:23.077729 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:23.077912 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:23.077917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.078586 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.078587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:23.079363 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:23.079364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.079738 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.079739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:23.079835 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.080408 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:23.080416 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:23.087685 (+ 7269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:23.089671 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:23.089678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:23.089680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:23.089904 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:23.090115 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:23.090116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:23.090116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:23.090217 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:23.090517 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:23.090521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:23.090635 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 12:26:23.090718 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:26:23.090837 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260928 12:26:23.091027 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.075910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:52457 (local address 127.7.19.126:34743)
0928 12:26:23.075975 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.075978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.076164 (+ 186us) server_negotiation.cc:407] Connection header received
0928 12:26:23.077248 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.077249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.077260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.077325 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.078001 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.078502 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.079469 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.079663 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.079727 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.087786 (+ 8059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:23.087787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:23.087788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:23.087797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:23.089582 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.089979 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.089981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.089981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.090046 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.090285 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.090286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.090286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.090447 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:23.090525 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:23.090766 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 12:26:23.090865 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:26:23.091528 13148 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34743
I20260928 12:26:23.091648 13148 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:23.091843 13148 heartbeater.cc:511] Master 127.7.19.126:34743 requested a full tablet report, sending...
I20260928 12:26:23.092248 12822 ts_manager.cc:194] Registered new tserver with Master: 502ff2f511a9407fbc47ef1180ba7cf6 (127.7.19.66:45983)
I20260928 12:26:23.092790 12822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:52457
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:23.111286 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34743
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:23.190416 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:23.190644 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:23.190680 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:23.191835 13153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:23.191900 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:23.191942 13153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:23.193459 13153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:23.193712 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:23.193933 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:23.198671 13161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:23.198689 13158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:23.198727 13159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:23.199012 13153 server_base.cc:1034] running on GCE node
I20260928 12:26:23.199177 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:23.199349 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:23.200469 13153 hybrid_clock.cc:648] HybridClock initialized: now 1790598383200458 us; error 27 us; skew 500 ppm
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.202893 13153 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:23.203300 13153 webserver.cc:460] Webserver started at http://127.7.19.67:42897/ using document root <none> and password file <none>
I20260928 12:26:23.203522 13153 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:23.203591 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:23.203737 13153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:23.204809 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "0cfd24aac76d4c82a9976ad44f663685"
format_stamp: "Formatted at 2026-09-28 12:26:23 on dist-test-slave-cb7x"
server_key: "490bf608f6de8933a1b6edc68c431f09"
server_key_iv: "1c6e413507252acc3f6f4043b066f4bf"
server_key_version: "encryptionkey@0"
I20260928 12:26:23.205201 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "0cfd24aac76d4c82a9976ad44f663685"
format_stamp: "Formatted at 2026-09-28 12:26:23 on dist-test-slave-cb7x"
server_key: "490bf608f6de8933a1b6edc68c431f09"
server_key_iv: "1c6e413507252acc3f6f4043b066f4bf"
server_key_version: "encryptionkey@0"
I20260928 12:26:23.206427 13153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:23.207083 13168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:23.207248 13153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:23.207342 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "0cfd24aac76d4c82a9976ad44f663685"
format_stamp: "Formatted at 2026-09-28 12:26:23 on dist-test-slave-cb7x"
server_key: "490bf608f6de8933a1b6edc68c431f09"
server_key_iv: "1c6e413507252acc3f6f4043b066f4bf"
server_key_version: "encryptionkey@0"
I20260928 12:26:23.207419 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:23.219831 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:23.222150 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:23.222312 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:23.222530 13153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:23.222832 13153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:23.222889 13153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:23.222929 13153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:23.222967 13153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:23.228662 13153 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:38829
I20260928 12:26:23.228770 13281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:38829 every 8 connection(s)
I20260928 12:26:23.229096 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:23.235523 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13153
I20260928 12:26:23.235623 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:23.235823 7245 external_mini_cluster.cc:1428] Setting key 6321dc22dcf4a3198b9cc7eca6693523
I20260928 12:26:23.246534 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.229840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:57705 (local address 127.7.19.126:34743)
0928 12:26:23.229890 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.229910 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.230099 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:26:23.231040 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.231042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.231053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.231111 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.231730 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.232238 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.233316 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.233518 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.233603 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.235717 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:23.235718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:23.235719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:23.235727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:23.245007 (+ 9280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.245513 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.245514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.245515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.245578 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.245852 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.245852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.245853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.245978 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:23.246055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:23.246346 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 12:26:23.246433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:26:23.246783 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.229798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34743 (local address 127.7.19.67:57705)
0928 12:26:23.229964 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:23.230009 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:23.230929 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:23.231183 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:23.231187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:23.231452 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:23.231636 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:23.231640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.232332 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.232333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:23.233214 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:23.233216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.233610 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.233611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:23.233731 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.234174 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:23.234182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:23.235599 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:23.245107 (+ 9508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:23.245117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:23.245120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:23.245410 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:23.245646 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:23.245646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:23.245647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:23.245771 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:23.246046 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:23.246053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:23.246211 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 12:26:23.246327 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 12:26:23.246515 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260928 12:26:23.247314 13282 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34743
I20260928 12:26:23.247424 13282 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:23.247633 13282 heartbeater.cc:511] Master 127.7.19.126:34743 requested a full tablet report, sending...
I20260928 12:26:23.248014 12821 ts_manager.cc:194] Registered new tserver with Master: 0cfd24aac76d4c82a9976ad44f663685 (127.7.19.67:38829)
I20260928 12:26:23.248399 12821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:57705
I20260928 12:26:23.258379 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/test-admin.keytab.
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.273615 7245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 12:26:23.275314 7245 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc' of type FILE
W20260928 12:26:23.275339 13289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:23.274503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.7.19.65:42413)
0928 12:26:23.274674 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.274679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.274689 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:23.274715 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.274717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.274730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.274795 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.275186 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35368: BlockingRecv error: recv got EOF from 127.0.0.1:35368 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.276309 7245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.277213 7245 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:23.284933 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.277717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.7.19.65:42413)
0928 12:26:23.277799 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.277802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.277812 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:23.277879 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.277880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.277892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.277987 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.278402 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.279182 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.279893 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.280216 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.280275 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.282293 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:23.282300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:23.282300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:23.282306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:23.283903 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.284183 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.284185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.284185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.284235 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.284376 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.284376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.284377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.284505 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:23.284613 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:23.284711 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 12:26:23.284781 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 12:26:23.286577 13289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:23.286111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.7.19.65:42413)
0928 12:26:23.286149 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.286152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.286182 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:26:23.286245 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.286246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.286253 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.286304 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.286527 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35398: BlockingRecv error: recv got EOF from 127.0.0.1:35398 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:26:23.286816 7245 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.287742 7245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.288573 7245 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:23.295325 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.289087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.7.19.65:42413)
0928 12:26:23.289164 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.289165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.289210 (+ 45us) server_negotiation.cc:407] Connection header received
0928 12:26:23.289256 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.289257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.289264 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.289302 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.289629 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.290115 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.290740 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.290921 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.291001 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.292991 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:23.292991 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:23.292992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:23.292997 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:23.294346 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.294654 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.294655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.294655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.294701 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.294853 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.294854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.294854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.294948 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:23.295015 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:23.295147 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 12:26:23.295198 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 12:26:23.297256 13289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:23.296535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.7.19.65:42413)
0928 12:26:23.296597 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.296600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.296632 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:26:23.296758 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.296759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.296767 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.296812 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.297199 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35414: BlockingRecv error: recv got EOF from 127.0.0.1:35414 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 12:26:23.297418 7245 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790598357699237-7245-0/krb5kdc/krb5cc' of type FILE
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.298319 7245 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:23.299173 7245 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[12777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:26:23.306143 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:23.299627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.7.19.65:42413)
0928 12:26:23.299692 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:23.299694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:23.299745 (+ 51us) server_negotiation.cc:407] Connection header received
0928 12:26:23.299827 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:23.299828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:23.299836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:23.299891 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:23.300259 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.300785 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.301454 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:23.301654 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:23.301735 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:23.303628 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:23.303629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:23.303630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:23.303636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:23.305028 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.305373 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.305374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.305375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.305428 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:23.305631 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:23.305632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:23.305632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:23.305733 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:23.305867 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:23.305986 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 12:26:23.306053 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:26:23.306744 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12884
I20260928 12:26:23.311899 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13019
I20260928 12:26:23.316497 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13153
I20260928 12:26:23.320495 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 12790
2026-09-28T12:26:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2766 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[13302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[13302](info): set up 2 sockets
Sep 28 12:26:23 dist-test-slave-cb7x krb5kdc[13302](info): commencing operation
krb5kdc: starting...
W20260928 12:26:24.356993 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab.
Entry for principal oryx/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab.
I20260928 12:26:24.476608 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45679
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab
--principal=oryx/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:45679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:24.555962 13315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:24.556159 13315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:24.556181 13315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:24.557389 13315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:24.557430 13315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:24.557444 13315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:24.557492 13315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:24.557508 13315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:24.558944 13315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:45679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab
--principal=oryx/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:24.559188 13315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:24.559412 13315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:24.564316 13323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:24.564325 13321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:24.564327 13320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:24.564371 13315 server_base.cc:1034] running on GCE node
I20260928 12:26:24.564827 13315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:24.565115 13315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:24.566275 13315 hybrid_clock.cc:648] HybridClock initialized: now 1790598384566260 us; error 28 us; skew 500 ppm
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:24.568650 13315 init.cc:377] Logged in from keytab as oryx/127.7.19.126@KRBTEST.COM (short username oryx)
I20260928 12:26:24.569106 13315 webserver.cc:460] Webserver started at http://127.7.19.126:46741/ using document root <none> and password file <none>
I20260928 12:26:24.569330 13315 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:24.569403 13315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:24.569553 13315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:24.570590 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "61c93fe733b14ced97fdb4d141471377"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "7125414111c63f438ab225e2197ca20b"
server_key_iv: "7c511f1c5fee0742ace730fe80d3c3c2"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.570999 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "61c93fe733b14ced97fdb4d141471377"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "7125414111c63f438ab225e2197ca20b"
server_key_iv: "7c511f1c5fee0742ace730fe80d3c3c2"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.572307 13315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:24.573123 13330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.573283 13315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.573410 13315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "61c93fe733b14ced97fdb4d141471377"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "7125414111c63f438ab225e2197ca20b"
server_key_iv: "7c511f1c5fee0742ace730fe80d3c3c2"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.573487 13315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:24.585392 13315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:24.587524 13315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:24.587687 13315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:24.591645 13315 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:45679
I20260928 12:26:24.591676 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:45679 every 8 connection(s)
I20260928 12:26:24.592027 13315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:24.593084 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:24.595213 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Bootstrap starting.
I20260928 12:26:24.595959 13383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:24.596261 13383 log.cc:824] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:24.597025 13383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: No bootstrap required, opened a new log
I20260928 12:26:24.598567 13383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } }
I20260928 12:26:24.598721 13383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:24.598765 13383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c93fe733b14ced97fdb4d141471377, State: Initialized, Role: FOLLOWER
I20260928 12:26:24.598955 13383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } }
I20260928 12:26:24.599052 13383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:24.599095 13383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:24.599144 13383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:24.599905 13383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } }
I20260928 12:26:24.600026 13383 leader_election.cc:302] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c93fe733b14ced97fdb4d141471377; no voters:
I20260928 12:26:24.600203 13383 leader_election.cc:288] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:24.600252 13386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:24.600438 13386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 LEADER]: Becoming Leader. State: Replica: 61c93fe733b14ced97fdb4d141471377, State: Running, Role: LEADER
I20260928 12:26:24.600446 13383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:24.600610 13386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } }
I20260928 12:26:24.601011 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } }
I20260928 12:26:24.601200 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13315
I20260928 12:26:24.601264 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:24.601301 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:24.601547 7245 external_mini_cluster.cc:1428] Setting key 5b0f6b6b3bec1569a0980fc833568821
I20260928 12:26:24.601193 13388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c93fe733b14ced97fdb4d141471377. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } }
I20260928 12:26:24.601696 13388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:24.601768 13395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:24.602296 13395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:24.603919 13395 catalog_manager.cc:1347] Generated new cluster ID: f8b2201c43f54977a0ebb4962f589b2f
I20260928 12:26:24.603969 13395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:24.611351 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:24.602263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.7.19.126:45679)
0928 12:26:24.602435 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:24.602445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:24.602483 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:26:24.602598 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:24.602605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:24.602662 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:24.602798 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:24.603523 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.604531 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.605345 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.605778 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.605904 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:24.608165 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:24.608169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:24.608170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:24.608181 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:24.609845 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:24.610215 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:24.610218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:24.610219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:24.610313 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:24.610519 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:24.610520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:24.610520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:24.610751 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:24.610911 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:24.611007 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:26:24.611145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260928 12:26:24.616477 13395 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:24.617182 13395 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:24.620730 13395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Generated new TSK 0
I20260928 12:26:24.620891 13395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab.
Entry for principal oryx/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab.
I20260928 12:26:24.647665 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45679
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:24.732613 13409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:24.732874 13409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:24.732898 13409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:24.734023 13409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:24.734062 13409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:24.734089 13409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:24.735491 13409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:24.735739 13409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:24.735961 13409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:24.740908 13417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:24.740916 13415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:24.740923 13414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:24.741191 13409 server_base.cc:1034] running on GCE node
I20260928 12:26:24.741369 13409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:24.741559 13409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:24.742688 13409 hybrid_clock.cc:648] HybridClock initialized: now 1790598384742679 us; error 26 us; skew 500 ppm
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:24.745169 13409 init.cc:377] Logged in from keytab as oryx/127.7.19.65@KRBTEST.COM (short username oryx)
I20260928 12:26:24.745566 13409 webserver.cc:460] Webserver started at http://127.7.19.65:43805/ using document root <none> and password file <none>
I20260928 12:26:24.745738 13409 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:24.745776 13409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:24.745860 13409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:24.746837 13409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "19a732fe7ce34d5f87f84824c9c012b1"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a57589936b16b2aefae5eaf1ab8b1df5"
server_key_iv: "f95406926e1b34e333e556f22423b322"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.747128 13409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "19a732fe7ce34d5f87f84824c9c012b1"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a57589936b16b2aefae5eaf1ab8b1df5"
server_key_iv: "f95406926e1b34e333e556f22423b322"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.748306 13409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:24.748998 13424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.749166 13409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.749261 13409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "19a732fe7ce34d5f87f84824c9c012b1"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a57589936b16b2aefae5eaf1ab8b1df5"
server_key_iv: "f95406926e1b34e333e556f22423b322"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.749341 13409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:24.759315 13409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:24.761593 13409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:24.761749 13409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:24.761965 13409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:24.762259 13409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:24.762311 13409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.762351 13409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:24.762379 13409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.767863 13409 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:40657
I20260928 12:26:24.767946 13537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:40657 every 8 connection(s)
I20260928 12:26:24.768249 13409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:24.773088 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13409
I20260928 12:26:24.773200 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:24.773414 7245 external_mini_cluster.cc:1428] Setting key 8f5fa3b9413c9884d0cfc0db81a137df
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:24.777519 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:24.768947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:49071 (local address 127.7.19.126:45679)
0928 12:26:24.769023 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:24.769026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:24.769233 (+ 207us) server_negotiation.cc:407] Connection header received
0928 12:26:24.769700 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:24.769701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:24.769712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:24.769763 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:24.770624 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.771111 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.772054 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.772243 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.772293 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:24.774502 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:24.774504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:24.774504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:24.774514 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:24.776044 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:24.776516 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:24.776517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:24.776517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:24.776563 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:24.776864 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:24.776865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:24.776865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:24.776970 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:24.777022 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:24.777386 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 12:26:24.777457 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:24.777817 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:24.768921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:49071)
0928 12:26:24.769118 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:24.769152 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:24.769621 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:24.769812 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:24.769815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:24.770356 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:24.770550 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:24.770554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.771187 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.771187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:24.771983 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:24.771985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.772310 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.772311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:24.772428 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:24.772937 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:24.772944 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:24.774405 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:24.776102 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:24.776109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:24.776111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:24.776435 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:24.776626 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:24.776628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:24.776628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:24.776796 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:24.777036 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:24.777059 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:24.777251 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 12:26:24.777364 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 12:26:24.777544 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260928 12:26:24.778332 13538 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45679
I20260928 12:26:24.778437 13538 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:24.778609 13538 heartbeater.cc:511] Master 127.7.19.126:45679 requested a full tablet report, sending...
I20260928 12:26:24.779199 13347 ts_manager.cc:194] Registered new tserver with Master: 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657)
I20260928 12:26:24.780176 13347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:49071
WARNING: no policy specified for oryx/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab.
Entry for principal oryx/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab.
I20260928 12:26:24.800093 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45679
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:24.878643 13543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:24.878831 13543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:24.878854 13543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:24.879902 13543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:24.879937 13543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:24.879963 13543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:24.881460 13543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:24.881642 13543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:24.881806 13543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:24.886562 13549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:24.886587 13548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:24.886592 13551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:24.886759 13543 server_base.cc:1034] running on GCE node
I20260928 12:26:24.886958 13543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:24.887192 13543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:24.888440 13543 hybrid_clock.cc:648] HybridClock initialized: now 1790598384888425 us; error 25 us; skew 500 ppm
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:24.891049 13543 init.cc:377] Logged in from keytab as oryx/127.7.19.66@KRBTEST.COM (short username oryx)
I20260928 12:26:24.891558 13543 webserver.cc:460] Webserver started at http://127.7.19.66:36191/ using document root <none> and password file <none>
I20260928 12:26:24.891808 13543 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:24.891875 13543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:24.892007 13543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:24.893091 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "750c8f17c9d749d794aeb3e6fd4a079f"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a481b8d105f258683bed09cc5dca79e3"
server_key_iv: "6a885cce68c46232262f9bc4c5da678e"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.893445 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "750c8f17c9d749d794aeb3e6fd4a079f"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a481b8d105f258683bed09cc5dca79e3"
server_key_iv: "6a885cce68c46232262f9bc4c5da678e"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.894719 13543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:24.895427 13558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.895598 13543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:24.895694 13543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "750c8f17c9d749d794aeb3e6fd4a079f"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a481b8d105f258683bed09cc5dca79e3"
server_key_iv: "6a885cce68c46232262f9bc4c5da678e"
server_key_version: "encryptionkey@0"
I20260928 12:26:24.895772 13543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:24.935199 13543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:24.937290 13543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:24.937466 13543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:24.937700 13543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:24.938027 13543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:24.938084 13543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.938133 13543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:24.938163 13543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:24.943732 13543 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:39469
I20260928 12:26:24.943789 13671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:39469 every 8 connection(s)
I20260928 12:26:24.944082 13543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:24.944793 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13543
I20260928 12:26:24.944893 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:24.945096 7245 external_mini_cluster.cc:1428] Setting key 8eab92fb2fd8724211c723e677e053c9
Sep 28 12:26:24 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:26:24.954831 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:24.944889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49497 (local address 127.7.19.126:45679)
0928 12:26:24.944958 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:24.944961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:24.945187 (+ 226us) server_negotiation.cc:407] Connection header received
0928 12:26:24.945814 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:24.945815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:24.945827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:24.945892 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:24.946811 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.947417 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.949052 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.949345 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.949411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:24.951831 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:24.951832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:24.951832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:24.951841 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:24.953567 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:24.954005 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:24.954007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:24.954007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:24.954067 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:24.954283 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:24.954284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:24.954284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:24.954379 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:24.954426 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:24.954673 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 12:26:24.954761 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:24.955082 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:24.944862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:49497)
0928 12:26:24.945041 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:24.945084 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:24.945653 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:24.945953 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:24.945956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:24.946522 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:24.946707 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:24.946712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.947525 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.947526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:24.948952 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:24.948956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:24.949444 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:24.949446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:24.949597 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:24.950133 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:24.950144 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:24.951708 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:24.953646 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:24.953652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:24.953654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:24.953931 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:24.954137 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:24.954137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:24.954137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:24.954230 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:24.954430 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:24.954434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:24.954544 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 12:26:24.954623 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:26:24.954836 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260928 12:26:24.955624 13672 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45679
I20260928 12:26:24.955771 13672 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:24.955960 13672 heartbeater.cc:511] Master 127.7.19.126:45679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.67@KRBTEST.COM" created.
I20260928 12:26:24.956413 13347 ts_manager.cc:194] Registered new tserver with Master: 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:24.956874 13347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:49497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab.
Entry for principal oryx/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab.
I20260928 12:26:24.972033 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45679
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:25.050086 13677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:25.050264 13677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:25.050287 13677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:25.051843 13677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:25.051899 13677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:25.051930 13677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:25.053726 13677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:25.053975 13677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:25.054251 13677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:25.059290 13685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.059450 13677 server_base.cc:1034] running on GCE node
W20260928 12:26:25.059291 13682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.059322 13683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.059708 13677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.059947 13677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.061100 13677 hybrid_clock.cc:648] HybridClock initialized: now 1790598385061088 us; error 25 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.063508 13677 init.cc:377] Logged in from keytab as oryx/127.7.19.67@KRBTEST.COM (short username oryx)
I20260928 12:26:25.063959 13677 webserver.cc:460] Webserver started at http://127.7.19.67:36111/ using document root <none> and password file <none>
I20260928 12:26:25.064190 13677 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.064256 13677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.064399 13677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:25.065492 13677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "ddb72fb9528747f59e66de91130dedc9"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "53e7acd88cd35df4a594aa4d8b64e063"
server_key_iv: "f7ff7692d314e306426a579e42ccc877"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.065845 13677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "ddb72fb9528747f59e66de91130dedc9"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "53e7acd88cd35df4a594aa4d8b64e063"
server_key_iv: "f7ff7692d314e306426a579e42ccc877"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.067119 13677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:26:25.067843 13692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.068037 13677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:25.068135 13677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "ddb72fb9528747f59e66de91130dedc9"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "53e7acd88cd35df4a594aa4d8b64e063"
server_key_iv: "f7ff7692d314e306426a579e42ccc877"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.068213 13677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.102902 13677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.104976 13677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:25.105136 13677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.105378 13677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:25.105688 13677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:25.105741 13677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.105782 13677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:25.105808 13677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.111466 13677 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:33749
I20260928 12:26:25.111548 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:33749 every 8 connection(s)
I20260928 12:26:25.111888 13677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:25.116437 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13677
I20260928 12:26:25.116539 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:25.116770 7245 external_mini_cluster.cc:1428] Setting key 79cd86f2a6f977de8fbe8067a14eca49
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:25.121562 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.112780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:53301 (local address 127.7.19.126:45679)
0928 12:26:25.112840 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.112842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.113055 (+ 213us) server_negotiation.cc:407] Connection header received
0928 12:26:25.113519 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.113520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.113537 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.113630 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.114553 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.115034 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.115978 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.116152 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.116202 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.118706 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.118707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.118708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.118714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.120148 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.120491 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.120491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.120492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.120558 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.120870 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.120871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.120871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.120977 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.121036 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.121431 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.121499 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:26:25.121718 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.112751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:53301)
0928 12:26:25.112920 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.112963 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.113438 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.113679 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.113682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.114282 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.114474 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.114478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.115109 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.115110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.115905 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.115906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.116220 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.116221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.116403 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.117156 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.117165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.118632 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.120215 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.120219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.120221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.120427 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.120622 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.120623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.120624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.120784 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.121091 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.121096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.121224 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.121352 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.121499 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 12:26:25.122162 13806 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45679
I20260928 12:26:25.122267 13806 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.122440 13806 heartbeater.cc:511] Master 127.7.19.126:45679 requested a full tablet report, sending...
I20260928 12:26:25.122789 13347 ts_manager.cc:194] Registered new tserver with Master: ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749)
I20260928 12:26:25.123155 13347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:53301
I20260928 12:26:25.128291 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.19.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.125.keytab.
Entry for principal oryx/127.7.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.125.keytab.
I20260928 12:26:25.153416 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:33567
--webserver_interface=127.7.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.125.keytab
--principal=oryx/127.7.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:45679,127.7.19.125:33567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:25.241899 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:25.242118 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:25.242144 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:25.243448 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:25.243494 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:25.243508 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:25.243522 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:25.243536 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:25.245226 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:45679,127.7.19.125:33567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.125.keytab
--principal=oryx/127.7.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:33567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
W20260928 12:26:25.245370 13811 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.19.126:45679,127.7.19.125:33567'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 12:26:25.245543 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:25.245832 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:25.251405 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.251364 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.251478 13811 server_base.cc:1034] running on GCE node
W20260928 12:26:25.251387 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.251906 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.252213 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.253348 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790598385253330 us; error 41 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.255805 13811 init.cc:377] Logged in from keytab as oryx/127.7.19.125@KRBTEST.COM (short username oryx)
I20260928 12:26:25.256242 13811 webserver.cc:460] Webserver started at http://127.7.19.125:33281/ using document root <none> and password file <none>
I20260928 12:26:25.256480 13811 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.256525 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.256708 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:25.257825 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/instance:
uuid: "a5838c5c62e34597bf56643d85bcb27a"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "d3c581a2148a669fcbc99e0303cbd3d4"
server_key_iv: "7d712e9a5c18bb746ebe405544909021"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.258179 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal/instance:
uuid: "a5838c5c62e34597bf56643d85bcb27a"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "d3c581a2148a669fcbc99e0303cbd3d4"
server_key_iv: "7d712e9a5c18bb746ebe405544909021"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.259476 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:25.260177 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.260344 13811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:25.260459 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
uuid: "a5838c5c62e34597bf56643d85bcb27a"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "d3c581a2148a669fcbc99e0303cbd3d4"
server_key_iv: "7d712e9a5c18bb746ebe405544909021"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.260542 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.290918 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.292894 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:25.293047 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.296584 13811 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.125:33567
I20260928 12:26:25.296624 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.125:33567 every 8 connection(s)
I20260928 12:26:25.296998 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
I20260928 12:26:25.297991 13879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:25.298314 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13811
I20260928 12:26:25.300401 13879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.125@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
I20260928 12:26:25.308784 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.301108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.7.19.126:45679)
0928 12:26:25.301154 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.301156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.301324 (+ 168us) server_negotiation.cc:407] Connection header received
0928 12:26:25.301395 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.301396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.301407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.301457 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.301935 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.302469 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.303290 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.303555 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.303623 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.305849 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.305851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.305851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.305857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.307424 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.307803 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.307804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.307805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.307872 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.308119 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.308120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.308120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.308265 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.308341 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.308598 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.308697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:26:25.308965 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.301083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:38462)
0928 12:26:25.301214 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.301250 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.301320 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.301479 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.301482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.301749 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.301847 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.301852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.302568 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.302569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.303193 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.303194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.303657 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.303658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.303737 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.304104 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.304113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.305769 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.307507 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.307512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.307515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.307726 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.307937 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.307937 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.307938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.308039 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.308332 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.308337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.308468 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.308577 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.308739 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260928 12:26:25.309559 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.298976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.7.19.125:33567)
0928 12:26:25.299164 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.299173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.299196 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:26:25.299604 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.299610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.299653 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.299763 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.300559 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.301853 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.302675 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.303092 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.303242 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.306331 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.306337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.306337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.306344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.307903 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.308286 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.308289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.308290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.308342 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.308552 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.308553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.308553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.308739 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.308906 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.309301 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.309374 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260928 12:26:25.309538 13879 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.125@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
I20260928 12:26:25.315325 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.309809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57650)
0928 12:26:25.309867 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.309878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.309919 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.310036 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.310037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.310173 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.310222 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.310223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.310816 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.310817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.311276 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.311277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.311640 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.311641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.311700 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.312009 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.312012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.313130 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.314376 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.314377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.314378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.314531 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.314724 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.314724 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.314725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.314775 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.315011 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.315012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.315045 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.315137 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.315223 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:26:25.315369 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.309856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.7.19.125:33567)
0928 12:26:25.309897 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.309900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.309908 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:25.309983 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.309984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.309991 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.310033 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.310304 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.310743 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.311353 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.311565 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.311632 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.313226 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.313227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.313228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.313233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.314294 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.314612 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.314613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.314613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.314662 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.314862 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.314862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.314863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.314950 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.315019 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.315161 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.315269 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":14}
I20260928 12:26:25.316555 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Bootstrap starting.
I20260928 12:26:25.317382 13879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:25.317598 13879 log.cc:824] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:25.318223 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: No bootstrap required, opened a new log
I20260928 12:26:25.319531 13879 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } }
I20260928 12:26:25.319654 13879 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:25.319677 13879 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5838c5c62e34597bf56643d85bcb27a, State: Initialized, Role: FOLLOWER
I20260928 12:26:25.319792 13879 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } }
I20260928 12:26:25.320041 13879 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:25.320034 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } } }
I20260928 12:26:25.320144 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:26:25.321718 13896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:26:25.321792 13896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:25.323297 13811 master_runner.cc:428] Detected that this master a5838c5c62e34597bf56643d85bcb27a is joining an existing cluster
I20260928 12:26:25.323335 13811 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.19.125:33567
I20260928 12:26:25.331486 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.324519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57656)
0928 12:26:25.324635 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.324645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.324737 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.324857 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.324858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.325177 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.325276 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.325278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.326165 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.326166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.327005 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.327006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.327459 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.327460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.327524 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.328036 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.328040 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.328455 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.330337 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.330338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.330594 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.330790 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.330790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.330869 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.331123 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.331124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.331156 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.331249 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.331363 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":83,"threads_started":1}
I20260928 12:26:25.331496 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.324546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.7.19.125:33567)
0928 12:26:25.324582 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.324583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.324714 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:26:25.324788 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.324789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.324798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.324841 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.325365 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.326067 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.327082 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.327377 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.327448 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.328545 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.328547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.328547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.328554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.330250 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330678 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.330678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.330679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.330724 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330937 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.330938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.330938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.331048 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.331085 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.331283 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.331415 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":18}
I20260928 12:26:25.331547 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.324335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.7.19.126:45679)
0928 12:26:25.324395 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.324397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.324693 (+ 296us) server_negotiation.cc:407] Connection header received
0928 12:26:25.324745 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.324746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.324757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.324809 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.325369 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.325853 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.326911 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.327219 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.327257 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.328561 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.328562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.328563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.328570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.330207 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330635 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.330636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.330637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.330693 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330875 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.330876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.330876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.331007 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.331081 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.331318 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.331432 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:25.332374 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.324244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:38464)
0928 12:26:25.324593 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.324604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.324690 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.324858 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.324859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.325171 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.325276 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.325278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.325924 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.325925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.326810 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.326812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.327330 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.327331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.327376 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.327878 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.327883 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.328454 (+ 571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.330310 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.330312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.330548 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.330748 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.330748 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.330749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.330806 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.331085 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.331086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.331143 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.332143 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.332242 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260928 12:26:25.335505 13885 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:25.335619 13885 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } }
I20260928 12:26:25.335843 13885 leader_election.cc:288] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679)
I20260928 12:26:25.336043 13357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5838c5c62e34597bf56643d85bcb27a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c93fe733b14ced97fdb4d141471377" is_pre_election: true
I20260928 12:26:25.336398 13828 leader_election.cc:302] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a5838c5c62e34597bf56643d85bcb27a; no voters: 61c93fe733b14ced97fdb4d141471377
I20260928 12:26:25.336540 13885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.126@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
I20260928 12:26:25.342288 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.334651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57662)
0928 12:26:25.334807 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.334838 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.334898 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.335032 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.335034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.335261 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.335311 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.335313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.336160 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.336193 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.336989 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.336992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.337452 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.337452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.337504 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.337963 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.337968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.339546 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.341224 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.341230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.341231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.341464 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.341666 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.341666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.341666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.341717 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.341990 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.341996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.342042 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.342119 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.342208 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260928 12:26:25.342314 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.334706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.7.19.125:33567)
0928 12:26:25.334746 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.334748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.334904 (+ 156us) server_negotiation.cc:407] Connection header received
0928 12:26:25.334956 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.334957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.334966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.335030 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.335382 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.336079 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.337062 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.337374 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.337447 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.339626 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.339628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.339628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.339635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.341144 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.341537 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.341538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.341539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.341594 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.341776 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.341777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.341777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.341897 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.341965 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.342160 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.342230 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:26:25.342968 13347 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.19.125:33567: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a"
member_type: NON_VOTER
last_known_addr {
host: "127.7.19.125"
port: 33567
}
attrs {
promote: true
}
}
dest_uuid: "61c93fe733b14ced97fdb4d141471377"
cas_config_opid_index: -1
I20260928 12:26:25.343273 13347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } }
I20260928 12:26:25.343736 13906 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a5838c5c62e34597bf56643d85bcb27a (127.7.19.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } }
I20260928 12:26:25.344044 13853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:25.344660 13906 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.19.125:33567
I20260928 12:26:25.344795 13853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Refusing update from remote peer 61c93fe733b14ced97fdb4d141471377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 12:26:25.344784 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } }
I20260928 12:26:25.344859 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:25.344939 13811 master.cc:537] Master@127.7.19.125:33567 shutting down...
I20260928 12:26:25.345563 13905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:26:25.346316 13333 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 12:26:25.346736 13333 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 -> Peer a5838c5c62e34597bf56643d85bcb27a (127.7.19.125:33567): Couldn't send request to peer a5838c5c62e34597bf56643d85bcb27a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:26:25.347517 13811 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:25.347591 13811 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:25.347630 13811 tablet_replica.cc:330] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: stopping tablet replica
I20260928 12:26:25.360260 13811 master.cc:552] Master@127.7.19.125:33567 shutdown complete.
I20260928 12:26:25.360321 13811 master_runner.cc:305] Clearing existing system tablet
I20260928 12:26:25.360461 13811 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.360512 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.361323 13811 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:25.361924 13910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.362103 13811 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.362162 13811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:25.362188 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
uuid: "a5838c5c62e34597bf56643d85bcb27a"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "d3c581a2148a669fcbc99e0303cbd3d4"
server_key_iv: "7d712e9a5c18bb746ebe405544909021"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.362972 13811 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:25.364647 13811 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:26:25.364738 13811 log.cc:1196] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 12:26:25.364952 13811 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Deleting consensus metadata
I20260928 12:26:25.365093 13811 master_runner.cc:315] Copying system tablet from 127.7.19.126:45679
I20260928 12:26:25.365700 13811 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.19.126:45679
I20260928 12:26:25.372552 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.366177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.7.19.126:45679)
0928 12:26:25.366248 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.366249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.366343 (+ 94us) server_negotiation.cc:407] Connection header received
0928 12:26:25.366373 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.366374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.366385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.366442 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.366801 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.367325 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.368413 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.368600 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.368664 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.369762 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.369763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.369763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.369771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.371376 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.371743 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.371744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.371744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.371798 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.372050 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.372051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.372052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.372179 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.372239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.372415 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.372482 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:26:25.372591 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.366079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:38480)
0928 12:26:25.366229 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.366241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.366314 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.366449 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.366449 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.366659 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.366721 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.366723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.367405 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.367407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.368313 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.368315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.368723 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.368724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.368794 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.369263 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.369269 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.369662 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.371450 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.371451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.371452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.371670 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.371862 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.371863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.371864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.371928 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.372252 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.372253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.372300 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.372378 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.372482 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260928 12:26:25.372926 13367 tablet_copy_service.cc:140] P 61c93fe733b14ced97fdb4d141471377: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a5838c5c62e34597bf56643d85bcb27a ({username='oryx', principal='oryx/127.7.19.125@KRBTEST.COM'} at 127.0.0.1:38480)
I20260928 12:26:25.373011 13367 tablet_copy_service.cc:161] P 61c93fe733b14ced97fdb4d141471377: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a5838c5c62e34597bf56643d85bcb27a at {username='oryx', principal='oryx/127.7.19.125@KRBTEST.COM'} at 127.0.0.1:38480: session id = a5838c5c62e34597bf56643d85bcb27a-00000000000000000000000000000000
I20260928 12:26:25.373844 13367 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:26:25.374428 13811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:26:25.375442 13811 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: tablet copy: Starting download of 0 data blocks...
I20260928 12:26:25.375598 13811 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: tablet copy: Starting download of 1 WAL segments...
I20260928 12:26:25.376612 13811 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:26:25.377430 13367 tablet_copy_service.cc:342] P 61c93fe733b14ced97fdb4d141471377: Request end of tablet copy session a5838c5c62e34597bf56643d85bcb27a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.19.125@KRBTEST.COM'} at 127.0.0.1:38480
I20260928 12:26:25.377490 13367 tablet_copy_service.cc:433] P 61c93fe733b14ced97fdb4d141471377: ending tablet copy session a5838c5c62e34597bf56643d85bcb27a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a5838c5c62e34597bf56643d85bcb27a
I20260928 12:26:25.379737 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:25.381053 13917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.381242 13811 server_base.cc:1034] running on GCE node
W20260928 12:26:25.381276 13918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.381489 13920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.381642 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.381783 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.382915 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790598385382901 us; error 32 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.384334 13811 init.cc:377] Logged in from keytab as oryx/127.7.19.125@KRBTEST.COM (short username oryx)
I20260928 12:26:25.384620 13811 webserver.cc:460] Webserver started at http://127.7.19.125:38183/ using document root <none> and password file <none>
I20260928 12:26:25.384781 13811 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.384835 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.385450 13811 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:25.385999 13927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.386117 13811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:25.386162 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
uuid: "a5838c5c62e34597bf56643d85bcb27a"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "d3c581a2148a669fcbc99e0303cbd3d4"
server_key_iv: "7d712e9a5c18bb746ebe405544909021"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.386232 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.400633 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.402470 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.405546 13811 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.125:33567
I20260928 12:26:25.405716 13979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.125:33567 every 8 connection(s)
I20260928 12:26:25.405920 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
I20260928 12:26:25.406638 13980 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:26:25.407074 13980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Bootstrap starting.
I20260928 12:26:25.409452 13980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:26:25.409698 13980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Bootstrap complete.
I20260928 12:26:25.409933 13980 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } }
I20260928 12:26:25.410002 13980 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a5838c5c62e34597bf56643d85bcb27a, State: Initialized, Role: LEARNER
I20260928 12:26:25.410028 13980 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } }
I20260928 12:26:25.410197 13980 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:25.410219 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } }
I20260928 12:26:25.410360 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: LEARNER
I20260928 12:26:25.412029 13995 catalog_manager.cc:1259] Loaded cluster ID: f8b2201c43f54977a0ebb4962f589b2f
I20260928 12:26:25.413604 13995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: loading cluster ID for follower catalog manager: success
I20260928 12:26:25.414312 13995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: acquiring CA information for follower catalog manager: success
I20260928 12:26:25.414588 13995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:26:25.418790 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.411211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.7.19.125:33567)
0928 12:26:25.411329 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.411331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.411393 (+ 62us) server_negotiation.cc:407] Connection header received
0928 12:26:25.411417 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.411418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.411428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.411527 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.412173 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.413027 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.414017 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.414320 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.414404 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.415917 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.415918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.415918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.415924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.417669 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.418059 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.418060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.418061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.418109 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.418313 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.418313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.418313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.418437 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.418512 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.418665 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.418731 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for oryx/127.7.19.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.124.keytab.
Entry for principal oryx/127.7.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.124.keytab.
I20260928 12:26:25.444161 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:33003
--webserver_interface=127.7.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.124.keytab
--principal=oryx/127.7.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:25.528656 13998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:25.528914 13998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:25.528950 13998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:25.530081 13998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:25.530151 13998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:25.530175 13998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:25.530205 13998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:25.530236 13998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:25.531744 13998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.124.keytab
--principal=oryx/127.7.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:33003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:25.532011 13998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:25.532230 13998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:25.537480 14004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.537406 14003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.537397 14006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.537443 13998 server_base.cc:1034] running on GCE node
I20260928 12:26:25.537976 13998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.538328 13998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.539529 13998 hybrid_clock.cc:648] HybridClock initialized: now 1790598385539522 us; error 37 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.542075 13998 init.cc:377] Logged in from keytab as oryx/127.7.19.124@KRBTEST.COM (short username oryx)
I20260928 12:26:25.542457 13998 webserver.cc:460] Webserver started at http://127.7.19.124:40427/ using document root <none> and password file <none>
I20260928 12:26:25.542695 13998 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.542739 13998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.542886 13998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:25.543881 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/instance:
uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "ef46d1dd879acac63dd32e58a3b5702b"
server_key_iv: "a720f8fbfe3251a3c12780627735c583"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.544231 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal/instance:
uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "ef46d1dd879acac63dd32e58a3b5702b"
server_key_iv: "a720f8fbfe3251a3c12780627735c583"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.545614 13998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:25.546376 14013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.546598 13998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:25.546696 13998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "ef46d1dd879acac63dd32e58a3b5702b"
server_key_iv: "a720f8fbfe3251a3c12780627735c583"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.546782 13998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.585798 13998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.587879 13998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:25.588023 13998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.592535 13998 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.124:33003
I20260928 12:26:25.592576 14065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.124:33003 every 8 connection(s)
I20260928 12:26:25.592971 13998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
I20260928 12:26:25.593940 14066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:25.595399 14066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:26:25.599509 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 13998
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.124@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:25.604566 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.596113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.7.19.126:45679)
0928 12:26:25.596173 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.596175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.596308 (+ 133us) server_negotiation.cc:407] Connection header received
0928 12:26:25.596801 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.596802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.596812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.596877 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.597427 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.597953 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.599016 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.599211 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.599301 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.601714 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.601716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.601716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.601724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.603125 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.603554 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.603556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.603556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.603640 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.603958 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.603959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.603959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.604063 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.604115 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.604448 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.604507 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:25.604843 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.596021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:38482)
0928 12:26:25.596182 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.596222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.596652 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.596928 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.596932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.597162 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.597357 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.597362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.598052 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.598053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.598928 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.598930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.599306 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.599307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.599449 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.599981 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.599990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.601573 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.603246 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.603251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.603253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.603461 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.603707 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.603708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.603708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.603877 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.604130 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.604135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.604302 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.604430 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.604595 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260928 12:26:25.605448 14066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.19.124@KRBTEST.COM
I20260928 12:26:25.608912 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.600312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.7.19.124:33003)
0928 12:26:25.600478 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.600509 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.600533 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:26:25.600554 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.600562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.600579 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.600651 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.601322 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.602331 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.603040 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.603344 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.603422 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.606063 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.606069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.606070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.606075 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.607732 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.608085 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.608088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.608088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.608146 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.608345 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.608346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.608346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.608478 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.608644 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.608708 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.608807 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.124@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
I20260928 12:26:25.612203 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.605676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57686)
0928 12:26:25.605779 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.605788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.605833 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.605963 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.605964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.606107 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.606160 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.606162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.606786 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.606787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.607608 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.607610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.608100 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.608101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.608174 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.608577 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.608581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.609914 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.611403 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.611404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.611405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.611550 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.611719 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.611719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.611720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.611777 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.611947 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.611947 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.611977 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.612027 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.612086 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 12:26:25.612210 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.605740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.7.19.125:33567)
0928 12:26:25.605785 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.605788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.605824 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:26:25.605875 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.605876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.605886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.605963 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.606236 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.606693 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.607694 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.608002 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.608074 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.609982 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.609983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.609983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.609990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.611332 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.611609 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.611610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.611610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.611656 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.611816 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.611816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.611817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.611899 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.611949 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.612040 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.612134 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:25.612634 14066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.124@KRBTEST.COM for oryx/127.7.19.124@KRBTEST.COM
I20260928 12:26:25.617857 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.612878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37748)
0928 12:26:25.612924 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.612934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.612972 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.613083 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.613083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.613179 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.613203 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.613204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.613701 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.613702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.614155 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.614156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.614457 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.614457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.614491 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.614880 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.614885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.615878 (+ 993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.617149 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.617149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.617150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.617305 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.617462 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.617463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.617463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.617508 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.617667 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.617667 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.617696 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.617745 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.617807 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:26:25.617921 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.612931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.7.19.124:33003)
0928 12:26:25.612977 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.612979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.612988 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:25.613033 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.613033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.613041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.613081 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.613242 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.613646 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.614219 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.614399 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.614449 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.615951 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.615952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.615952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.615958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.617087 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.617364 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.617365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.617365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.617407 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.617546 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.617546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.617546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.617620 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.617662 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.617759 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.617848 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":30}
I20260928 12:26:25.619045 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Bootstrap starting.
I20260928 12:26:25.619812 14066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:25.620079 14066 log.cc:824] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:25.620826 14066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: No bootstrap required, opened a new log
I20260928 12:26:25.622668 14066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } }
I20260928 12:26:25.622804 14066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:25.622851 14066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ceaa3e4b3a4897a221c1a5e2898179, State: Initialized, Role: FOLLOWER
I20260928 12:26:25.623003 14066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } }
I20260928 12:26:25.623270 14066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:25.623231 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } } }
I20260928 12:26:25.623381 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:26:25.624919 14082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:26:25.624982 14082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:25.626991 13998 master_runner.cc:428] Detected that this master 46ceaa3e4b3a4897a221c1a5e2898179 is joining an existing cluster
I20260928 12:26:25.627027 13998 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.19.124:33003
I20260928 12:26:25.634682 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.628274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.7.19.125:33567)
0928 12:26:25.628312 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.628314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.628322 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:25.628340 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.628341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.628349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.628415 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.628815 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.629511 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.630607 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.630898 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.630937 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.631912 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.631913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.631914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.631920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.633540 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.633888 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.633889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.633889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.633946 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.634134 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.634135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.634135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.634259 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.634329 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.634515 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.634586 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:25.634799 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.627996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57688)
0928 12:26:25.628241 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.628249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.628285 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.628432 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.628440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.628636 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.628734 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.628736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.629593 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.629593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.630510 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.630512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.631025 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.631025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.631093 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.631524 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.631528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.631827 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.633624 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.633625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.633625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.633810 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.634015 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.634016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.634016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.634072 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.634356 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.634357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.634407 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.634475 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.634547 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":49,"thread_start_us":62,"threads_started":1}
I20260928 12:26:25.656504 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.628147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37760)
0928 12:26:25.628764 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.628774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.628840 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.628982 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.628982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.629174 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.629225 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.629226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.630057 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.630058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.630888 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.630890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.631333 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.631333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.631409 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.653254 (+ 21845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.653259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.653727 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.655424 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.655425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.655426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.655587 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.655787 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.655788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.655788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.655842 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.656123 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.656123 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.656161 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.656249 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.656376 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":53,"threads_started":1}
I20260928 12:26:25.656507 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.628295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.7.19.124:33003)
0928 12:26:25.628328 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.628330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.628820 (+ 490us) server_negotiation.cc:407] Connection header received
0928 12:26:25.628902 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.628903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.628910 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.628935 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.629300 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.629972 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.630975 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.631238 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.631273 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.653828 (+ 22555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.653830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.653830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.653838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.655324 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.655670 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.655671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.655672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.655723 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.655911 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.655912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.655912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.656038 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.656102 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.656277 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.656425 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":17}
I20260928 12:26:25.656803 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.627949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.7.19.126:45679)
0928 12:26:25.627992 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.627995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.628114 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:26:25.628182 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.628182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.628192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.628243 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.628734 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.629494 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.630608 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.630898 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.630946 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.653920 (+ 22974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.653921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.653922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.653929 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.655642 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.655995 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.655996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.655997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.656046 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.656231 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.656233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.656233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.656368 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.656438 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.656613 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.656711 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:25.657111 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.627924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:38498)
0928 12:26:25.628062 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.628071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.628124 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.628287 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.628287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.628531 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.628596 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.628598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.629593 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.629593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.630510 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.630512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.631023 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.631024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.631093 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.653413 (+ 22320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.653418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.653742 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.655737 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.655738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.655739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.655935 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.656116 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.656118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.656118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.656168 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.656466 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.656467 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.656506 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.656914 (+ 408us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.657011 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":27,"thread_start_us":94,"threads_started":1}
I20260928 12:26:25.660020 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13409
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598384, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.126@KRBTEST.COM for oryx/127.7.19.124@KRBTEST.COM
I20260928 12:26:25.665505 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.659201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.7.19.124:33003)
0928 12:26:25.659255 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.659257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.659265 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:25.659316 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.659317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.659325 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.659379 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.659641 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.660253 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.661079 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.661279 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.661360 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.663047 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.663049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.663049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.663056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.664387 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.664778 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.664779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.664779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.664831 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.665017 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.665017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.665018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.665139 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.665192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.665355 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.665418 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:26:25.665510 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.659144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37768)
0928 12:26:25.659198 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.659209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.659254 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.659383 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.659383 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.659526 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.659563 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.659565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.660330 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.660331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.660987 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.660989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.661361 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.661362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.661425 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.661823 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.661828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.662965 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.664488 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.664489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.664489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.664653 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.664895 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.664896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.664897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.664957 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.665214 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.665214 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.665255 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.665323 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.665420 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:26:25.666277 13347 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.19.124:33003: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
member_type: NON_VOTER
last_known_addr {
host: "127.7.19.124"
port: 33003
}
attrs {
promote: true
}
}
dest_uuid: "61c93fe733b14ced97fdb4d141471377"
cas_config_opid_index: 5
I20260928 12:26:25.666541 13347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } }
I20260928 12:26:25.666865 13905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.667366 14040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:25.667577 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:40657
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:45679
--webserver_port=43805
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:26:25.667884 13905 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.19.124:33003
I20260928 12:26:25.667951 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.668026 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:25.668221 13998 master.cc:537] Master@127.7.19.124:33003 shutting down...
I20260928 12:26:25.668318 14040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Refusing update from remote peer 61c93fe733b14ced97fdb4d141471377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:26:25.668599 13906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:26:25.669315 13333 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 -> Peer 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124:33003): Couldn't send request to peer 46ceaa3e4b3a4897a221c1a5e2898179. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 12:26:25.672127 13998 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:25.672224 13998 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:25.672283 13998 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: stopping tablet replica
I20260928 12:26:25.675045 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.667002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57700)
0928 12:26:25.669771 (+ 2769us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.669782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.669839 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.671687 (+ 1848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.671688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.671690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:25.671728 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.671730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.672700 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.672701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.673904 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.673907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.674804 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.674805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.674868 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.674889 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.674938 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.674975 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2752}
I20260928 12:26:25.675235 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.667144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.7.19.125:33567)
0928 12:26:25.667260 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.667262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.671100 (+ 3838us) server_negotiation.cc:407] Connection header received
0928 12:26:25.671131 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.671132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.671156 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.671223 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:25.671812 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.672578 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.674001 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.674633 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.674702 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.674772 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.675136 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.675172 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 12:26:25.675541 13954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 LEARNER]: Refusing update from remote peer 61c93fe733b14ced97fdb4d141471377: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 12:26:25.675799 13906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 12:26:25.676337 13954 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.677685 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c93fe733b14ced97fdb4d141471377. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.677760 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: LEARNER
I20260928 12:26:25.677794 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: NON_VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: true } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.677834 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: LEARNER
I20260928 12:26:25.678128 13905 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: attempting to promote NON_VOTER a5838c5c62e34597bf56643d85bcb27a to VOTER
I20260928 12:26:25.678351 13905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } }
I20260928 12:26:25.678879 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.678745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37772)
0928 12:26:25.678815 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.678852 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:33003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 12:26:25.679014 13954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 LEARNER]: Refusing update from remote peer 61c93fe733b14ced97fdb4d141471377: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 12:26:25.679260 14094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 12:26:25.679661 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.679387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37776)
0928 12:26:25.679607 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.679633 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:33003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
W20260928 12:26:25.679855 13333 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 -> Peer 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124:33003): Couldn't send request to peer 46ceaa3e4b3a4897a221c1a5e2898179. Status: Network error: Client connection negotiation failed: client connection to 127.7.19.124:33003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:26:25.680032 13906 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5838c5c62e34597bf56643d85bcb27a (127.7.19.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.680923 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.681002 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:25.680953 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.681085 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:25.681164 13954 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5838c5c62e34597bf56643d85bcb27a (127.7.19.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.682152 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.682240 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:26:25.685220 13998 master.cc:552] Master@127.7.19.124:33003 shutdown complete.
I20260928 12:26:25.685261 13998 master_runner.cc:305] Clearing existing system tablet
I20260928 12:26:25.685398 13998 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.685442 13998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.686053 13998 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:25.686610 14096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.686759 13998 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.686825 13998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:25.686849 13998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "ef46d1dd879acac63dd32e58a3b5702b"
server_key_iv: "a720f8fbfe3251a3c12780627735c583"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.688521 13998 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:25.690645 13998 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 12:26:25.690730 13998 log.cc:1196] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 12:26:25.690935 13998 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Deleting consensus metadata
I20260928 12:26:25.691219 13998 master_runner.cc:315] Copying system tablet from 127.7.19.126:45679
I20260928 12:26:25.691821 13998 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: tablet copy: Beginning tablet copy session from remote peer at address 127.7.19.126:45679
I20260928 12:26:25.703692 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.692141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.7.19.126:45679)
0928 12:26:25.692180 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.692182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.696846 (+ 4664us) server_negotiation.cc:407] Connection header received
0928 12:26:25.696951 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.696952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.696964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.697040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.697397 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.698170 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.699272 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.699555 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.699620 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.700764 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.700765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.700765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.700772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.702505 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.702887 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.702888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.702889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.702952 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.703167 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.703168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.703168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.703295 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.703362 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.703531 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.703605 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:25.703693 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.692083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:38508)
0928 12:26:25.696770 (+ 4687us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.696782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.696874 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.697051 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.697052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.697260 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.697316 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.697318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.698254 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.698255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.699176 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.699179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.699633 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.699634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.699690 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.700192 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.700197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.700659 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.702590 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.702591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.702591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.702802 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.703029 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.703030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.703030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.703098 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.703367 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.703367 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.703420 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.703497 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.703594 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4663,"thread_start_us":64,"threads_started":1}
I20260928 12:26:25.704326 13367 tablet_copy_service.cc:140] P 61c93fe733b14ced97fdb4d141471377: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 46ceaa3e4b3a4897a221c1a5e2898179 ({username='oryx', principal='oryx/127.7.19.124@KRBTEST.COM'} at 127.0.0.1:38508)
I20260928 12:26:25.704393 13367 tablet_copy_service.cc:161] P 61c93fe733b14ced97fdb4d141471377: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 46ceaa3e4b3a4897a221c1a5e2898179 at {username='oryx', principal='oryx/127.7.19.124@KRBTEST.COM'} at 127.0.0.1:38508: session id = 46ceaa3e4b3a4897a221c1a5e2898179-00000000000000000000000000000000
I20260928 12:26:25.705030 13367 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 12:26:25.705448 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 12:26:25.706584 13998 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: tablet copy: Starting download of 0 data blocks...
I20260928 12:26:25.706728 13998 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: tablet copy: Starting download of 1 WAL segments...
I20260928 12:26:25.707437 13998 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 12:26:25.708276 13367 tablet_copy_service.cc:342] P 61c93fe733b14ced97fdb4d141471377: Request end of tablet copy session 46ceaa3e4b3a4897a221c1a5e2898179-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.19.124@KRBTEST.COM'} at 127.0.0.1:38508
I20260928 12:26:25.708328 13367 tablet_copy_service.cc:433] P 61c93fe733b14ced97fdb4d141471377: ending tablet copy session 46ceaa3e4b3a4897a221c1a5e2898179-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 46ceaa3e4b3a4897a221c1a5e2898179
I20260928 12:26:25.710784 13998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:25.711938 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.712177 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.712496 13998 server_base.cc:1034] running on GCE node
W20260928 12:26:25.713337 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.713564 13998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.713698 13998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.714838 13998 hybrid_clock.cc:648] HybridClock initialized: now 1790598385714825 us; error 28 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.716286 13998 init.cc:377] Logged in from keytab as oryx/127.7.19.124@KRBTEST.COM (short username oryx)
I20260928 12:26:25.716533 13998 webserver.cc:460] Webserver started at http://127.7.19.124:41311/ using document root <none> and password file <none>
I20260928 12:26:25.716645 13998 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.716708 13998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.717402 13998 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:25.717957 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.718125 13998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:25.718189 13998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "ef46d1dd879acac63dd32e58a3b5702b"
server_key_iv: "a720f8fbfe3251a3c12780627735c583"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.718242 13998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.740639 13998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.742522 13998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.745628 13998 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.124:33003
I20260928 12:26:25.745731 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.124:33003 every 8 connection(s)
I20260928 12:26:25.745949 13998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
I20260928 12:26:25.746803 14166 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:26:25.747280 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Bootstrap starting.
I20260928 12:26:25.750258 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:26:25.750494 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: Bootstrap complete.
I20260928 12:26:25.750890 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } }
I20260928 12:26:25.751003 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46ceaa3e4b3a4897a221c1a5e2898179, State: Initialized, Role: LEARNER
I20260928 12:26:25.751047 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } }
I20260928 12:26:25.751246 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:25.751264 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:25.751484 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: This master's current role is: LEARNER
I20260928 12:26:25.752875 14180 catalog_manager.cc:1259] Loaded cluster ID: f8b2201c43f54977a0ebb4962f589b2f
I20260928 12:26:25.752960 14180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: loading cluster ID for follower catalog manager: success
I20260928 12:26:25.754105 14180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: acquiring CA information for follower catalog manager: success
I20260928 12:26:25.754408 14180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 12:26:25.770535 14091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:25.770744 14091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:25.770785 14091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:25.771994 14091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:25.772056 14091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:25.772100 14091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:25.773608 14091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:40657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=43805
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:25.773854 14091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:25.774076 14091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:25.779570 14186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.779579 14185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.779608 14188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.779847 14091 server_base.cc:1034] running on GCE node
I20260928 12:26:25.780048 14091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.780280 14091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.781415 14091 hybrid_clock.cc:648] HybridClock initialized: now 1790598385781412 us; error 53 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.784210 14091 init.cc:377] Logged in from keytab as oryx/127.7.19.65@KRBTEST.COM (short username oryx)
I20260928 12:26:25.784706 14091 webserver.cc:460] Webserver started at http://127.7.19.65:43805/ using document root <none> and password file <none>
I20260928 12:26:25.784955 14091 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.785046 14091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.786430 14091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:25.787075 14195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.787235 14091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:25.787317 14091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "19a732fe7ce34d5f87f84824c9c012b1"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a57589936b16b2aefae5eaf1ab8b1df5"
server_key_iv: "f95406926e1b34e333e556f22423b322"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.787617 14091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.804193 14091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.806007 14091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:25.806113 14091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.806279 14091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:25.806505 14091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:25.806532 14091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.806553 14091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:25.806566 14091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.812616 14091 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:40657
I20260928 12:26:25.812777 14308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:40657 every 8 connection(s)
I20260928 12:26:25.813067 14091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.124@KRBTEST.COM
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
I20260928 12:26:25.822928 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 14091
I20260928 12:26:25.823093 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13543
I20260928 12:26:25.824707 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.813762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:45779 (local address 127.7.19.126:45679)
0928 12:26:25.813818 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.813820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.814057 (+ 237us) server_negotiation.cc:407] Connection header received
0928 12:26:25.815050 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.815051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.815063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.815127 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.816002 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.816488 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.817608 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.817896 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.817962 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.820745 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.820752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.820752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.820758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.822203 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.822674 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.822675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.822676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.822742 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.823770 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.823771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.823771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.823937 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.823982 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.824454 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.824541 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:25.825412 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.814209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:41043 (local address 127.7.19.125:33567)
0928 12:26:25.814251 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.814253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.815184 (+ 931us) server_negotiation.cc:407] Connection header received
0928 12:26:25.815235 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.815236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.815248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.815300 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.815970 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.816466 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.817746 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.818049 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.818141 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.821934 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.821935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.821935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.821943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.823678 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.824086 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.824087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.824088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.824142 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.824354 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.824355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.824355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.824655 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.824728 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.825196 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.825279 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:26:25.826296 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.814053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.7.19.65:41043)
0928 12:26:25.815127 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.815137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.815185 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.815305 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.815306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.815678 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.815875 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.815877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.816573 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.816574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.817678 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.817680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.818132 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.818133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.818196 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.819601 (+ 1405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.819606 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.821800 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.823770 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.823771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.823771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.824003 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.824211 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.824212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.824212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.824270 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.825032 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.825032 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.825080 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.825819 (+ 739us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.826025 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
I20260928 12:26:25.826777 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.813770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.7.19.65:50233)
0928 12:26:25.813953 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.813988 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.814943 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.815196 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.815200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.815703 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.815870 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.815875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.817693 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.817694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.818435 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.818438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.818795 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.818796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.818855 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.819228 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.819232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.821740 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.825647 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.825648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.825648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.825863 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.826122 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.826123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.826124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.826180 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.826547 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.826548 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.826584 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.826649 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.826713 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":12,"thread_start_us":91,"threads_started":1}
I20260928 12:26:25.826974 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.813770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:45779)
0928 12:26:25.813945 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.813986 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.814960 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.815196 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.815201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.815693 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.815870 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.815875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.816581 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.816582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.817515 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.817516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.818021 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.818022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.818151 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.818802 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.818812 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.820637 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.822275 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.822281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.822283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.822587 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.823548 (+ 961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.823549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.823550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.823686 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.824029 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.824036 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.824234 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.825932 (+ 1698us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.826292 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":292,"thread_start_us":118,"threads_started":1}
I20260928 12:26:25.826967 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.814012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:50233 (local address 127.7.19.124:33003)
0928 12:26:25.814123 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.814125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.814135 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:25.815049 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.815051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.815065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.815310 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.816582 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.817613 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.818516 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.818725 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.818798 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.821840 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.821841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.821842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.821848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.825559 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.826001 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.826002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.826002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.826058 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.826317 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.826318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.826318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.826459 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.826522 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.826714 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.826812 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260928 12:26:25.827409 14310 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45679
I20260928 12:26:25.827502 14310 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.827667 14310 heartbeater.cc:511] Master 127.7.19.126:45679 requested a full tablet report, sending...
I20260928 12:26:25.828105 13347 ts_manager.cc:194] Re-registered known tserver with Master: 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657)
I20260928 12:26:25.828473 13347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:45779
I20260928 12:26:25.829468 14311 heartbeater.cc:347] Connected to a master server at 127.7.19.125:33567
I20260928 12:26:25.829525 14311 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.829591 14311 heartbeater.cc:511] Master 127.7.19.125:33567 requested a full tablet report, sending...
I20260928 12:26:25.829948 13944 ts_manager.cc:194] Registered new tserver with Master: 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657)
I20260928 12:26:25.829982 14309 heartbeater.cc:347] Connected to a master server at 127.7.19.124:33003
I20260928 12:26:25.830018 14309 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.830068 14309 heartbeater.cc:511] Master 127.7.19.124:33003 requested a full tablet report, sending...
I20260928 12:26:25.830307 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:39469
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:45679
--webserver_port=36191
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:26:25.830360 14130 ts_manager.cc:194] Registered new tserver with Master: 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657)
W20260928 12:26:25.909837 14317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:25.910050 14317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:25.910089 14317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:25.911365 14317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:25.911425 14317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:25.911484 14317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:25.913062 14317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=36191
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:25.913313 14317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:25.913534 14317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:25.918412 14317 server_base.cc:1034] running on GCE node
W20260928 12:26:25.918396 14322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.918476 14325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:25.918398 14323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:25.918795 14317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:25.919035 14317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:25.920197 14317 hybrid_clock.cc:648] HybridClock initialized: now 1790598385920180 us; error 33 us; skew 500 ppm
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:25.922552 14317 init.cc:377] Logged in from keytab as oryx/127.7.19.66@KRBTEST.COM (short username oryx)
I20260928 12:26:25.922999 14317 webserver.cc:460] Webserver started at http://127.7.19.66:36191/ using document root <none> and password file <none>
I20260928 12:26:25.923231 14317 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:25.923313 14317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:25.924589 14317 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:25.925163 14332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.925340 14317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.925426 14317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "750c8f17c9d749d794aeb3e6fd4a079f"
format_stamp: "Formatted at 2026-09-28 12:26:24 on dist-test-slave-cb7x"
server_key: "a481b8d105f258683bed09cc5dca79e3"
server_key_iv: "6a885cce68c46232262f9bc4c5da678e"
server_key_version: "encryptionkey@0"
I20260928 12:26:25.925748 14317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:25.942389 14317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:25.944181 14317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:25.944344 14317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:25.944562 14317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:25.944914 14317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:25.944977 14317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.945017 14317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:25.945045 14317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:25.951181 14317 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:39469
I20260928 12:26:25.951232 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:39469 every 8 connection(s)
I20260928 12:26:25.951591 14317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:25.954442 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 14317
I20260928 12:26:25.954586 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13677
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:25.960395 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:33749
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:45679
--webserver_port=36111
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:26:25.962855 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.952591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:53555 (local address 127.7.19.126:45679)
0928 12:26:25.952631 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.952633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.952642 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:25.953288 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.953289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.953301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.953334 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.954186 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.954945 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.956611 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.956996 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.957062 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.959609 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.959610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.959611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.959618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.961625 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.961986 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.961987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.961988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.962041 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.962268 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.962269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.962269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.962395 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.962438 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.962676 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.962754 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:25.963065 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.952595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.7.19.66:33665)
0928 12:26:25.952778 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.952787 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.953175 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.953421 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.953426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.953897 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.954079 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.954085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.955332 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.955333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.956521 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.956523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.957039 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.957040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.957167 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.957630 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.957637 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.959083 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.960996 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.961002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.961004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.961305 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.961511 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.961512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.961512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.961768 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.962337 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.962343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.962518 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.962628 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.962804 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":42,"thread_start_us":57,"threads_started":1}
Sep 28 12:26:25 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598385, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.124@KRBTEST.COM
I20260928 12:26:25.963537 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.952765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:33665 (local address 127.7.19.125:33567)
0928 12:26:25.952814 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.952816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.952825 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:25.953287 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.953289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.953301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.953334 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.954184 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.954936 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.956742 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.956944 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.957032 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.959189 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.959191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.959191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.959199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.960900 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.961385 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.961387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.961387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.961441 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.962093 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.962094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.962095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.962242 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.962309 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.963303 (+ 994us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.963389 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:25.963817 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.952308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:53555)
0928 12:26:25.952497 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.952540 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.953172 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.953422 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.953426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.953865 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.954079 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.954085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.955204 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.955204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.956521 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.956523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.957114 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.957115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.957197 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.957945 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.957951 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.959523 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.961701 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.961702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.961702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.961909 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.962126 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.962127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.962127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.962188 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.962486 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.962486 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.962525 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.963421 (+ 896us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.963515 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":92,"thread_start_us":114,"threads_started":1}
I20260928 12:26:25.963845 14448 heartbeater.cc:347] Connected to a master server at 127.7.19.125:33567
I20260928 12:26:25.964290 14448 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.964512 14448 heartbeater.cc:511] Master 127.7.19.125:33567 requested a full tablet report, sending...
I20260928 12:26:25.964673 14447 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45679
I20260928 12:26:25.964767 14447 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.965008 13944 ts_manager.cc:194] Registered new tserver with Master: 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:25.965487 14447 heartbeater.cc:511] Master 127.7.19.126:45679 requested a full tablet report, sending...
I20260928 12:26:25.965734 13347 ts_manager.cc:194] Re-registered known tserver with Master: 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:25.966262 13347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:53555
I20260928 12:26:25.966738 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.952288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49883 (local address 127.7.19.124:33003)
0928 12:26:25.952330 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:25.952333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:25.952637 (+ 304us) server_negotiation.cc:407] Connection header received
0928 12:26:25.953281 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:25.953282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:25.953294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:25.953349 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:25.954172 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.954980 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.961153 (+ 6173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.961496 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.961541 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.963722 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:25.963723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:25.963723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:25.963730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:25.965398 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.965806 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.965807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.965808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.965861 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:25.966087 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:25.966088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:25.966088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:25.966207 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:25.966272 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:25.966512 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 12:26:25.966605 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:26:25.967089 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:25.952308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.7.19.66:49883)
0928 12:26:25.952566 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:25.952578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:25.953175 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:25.953421 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:25.953425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:25.953861 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:25.954078 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.954082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.960152 (+ 6070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.960153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.961048 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:25.961050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:25.961619 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:25.961621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:25.961692 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:25.962166 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:25.962170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:25.963620 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:25.965474 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.965475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.965476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.965725 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.965946 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:25.965947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:25.965947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:25.966014 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:25.966310 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:25.966311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:25.966360 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 12:26:25.966827 (+ 467us) client_negotiation.cc:240] Negotiation successful
0928 12:26:25.966948 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":23,"thread_start_us":90,"threads_started":1}
I20260928 12:26:25.967414 14446 heartbeater.cc:347] Connected to a master server at 127.7.19.124:33003
I20260928 12:26:25.967473 14446 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:25.967551 14446 heartbeater.cc:511] Master 127.7.19.124:33003 requested a full tablet report, sending...
I20260928 12:26:25.967829 14130 ts_manager.cc:194] Registered new tserver with Master: 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
W20260928 12:26:26.042902 14453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:26.043108 14453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:26.043131 14453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:26.044205 14453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:26.044238 14453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:26.044265 14453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:26.045656 14453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:33749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=36111
--tserver_master_addrs=127.7.19.126:45679,127.7.19.125:33567,127.7.19.124:33003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:26.045835 14453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:26.046010 14453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:26.050895 14461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:26.050896 14458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:26.050980 14453 server_base.cc:1034] running on GCE node
W20260928 12:26:26.050949 14459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:26.051294 14453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:26.051538 14453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:26.052717 14453 hybrid_clock.cc:648] HybridClock initialized: now 1790598386052704 us; error 27 us; skew 500 ppm
Sep 28 12:26:26 dist-test-slave-cb7x krb5kdc[13302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598386, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:26.055685 14453 init.cc:377] Logged in from keytab as oryx/127.7.19.67@KRBTEST.COM (short username oryx)
I20260928 12:26:26.056134 14453 webserver.cc:460] Webserver started at http://127.7.19.67:36111/ using document root <none> and password file <none>
I20260928 12:26:26.056362 14453 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:26.056419 14453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:26.057806 14453 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:26.058362 14468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:26.058540 14453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:26.058636 14453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "ddb72fb9528747f59e66de91130dedc9"
format_stamp: "Formatted at 2026-09-28 12:26:25 on dist-test-slave-cb7x"
server_key: "53e7acd88cd35df4a594aa4d8b64e063"
server_key_iv: "f7ff7692d314e306426a579e42ccc877"
server_key_version: "encryptionkey@0"
I20260928 12:26:26.058983 14453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:26.070636 14453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:26.072606 14453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:26.072801 14453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:26.073035 14453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:26.073329 14453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:26.073381 14453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:26.073421 14453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:26.073449 14453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:26.079447 14453 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:33749
I20260928 12:26:26.079493 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:33749 every 8 connection(s)
I20260928 12:26:26.079838 14453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:26.085915 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 14453
Sep 28 12:26:26 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598386, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.124@KRBTEST.COM
Sep 28 12:26:26 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598386, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.125@KRBTEST.COM
Sep 28 12:26:26 dist-test-slave-cb7x krb5kdc[13302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598386, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:26.092711 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.080834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.7.19.67:55807)
0928 12:26:26.080975 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.080984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.081815 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.082249 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.082250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.082596 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:26.082812 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.082818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.083869 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.083870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.084721 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.084722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.085166 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.085166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.085226 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.085794 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:26.085799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:26.087607 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:26.090946 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:26.090952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:26.090955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:26.091273 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:26.091500 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:26.091501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:26.091501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:26.091628 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:26.091963 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:26.091969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:26.092132 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.092239 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.092419 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":10,"thread_start_us":59,"threads_started":1}
I20260928 12:26:26.093039 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.080908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:55807 (local address 127.7.19.124:33003)
0928 12:26:26.080962 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.080964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.081033 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:26:26.082172 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.082174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.082185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.082234 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.082919 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.083790 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.084778 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.085057 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.085116 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.088533 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.088535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.088535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.088541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.090849 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.091371 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.091373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.091373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.091429 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.091704 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.091705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.091706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.091857 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.091934 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.092843 (+ 909us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.092935 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:26:26.093606 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.080530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.7.19.67:59405)
0928 12:26:26.080722 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.080754 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.081818 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.082202 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.082202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.082593 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:26.082811 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.082816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.083650 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.083651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.084452 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.084454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.084949 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.084949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.085080 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.085549 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:26.085557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:26.088006 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:26.092061 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:26.092062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:26.092062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:26.092269 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:26.092744 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:26.092745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:26.092745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:26.092818 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:26.093242 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:26.093243 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:26.093296 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.093367 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.093492 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":25,"thread_start_us":111,"threads_started":1}
I20260928 12:26:26.093672 14582 heartbeater.cc:347] Connected to a master server at 127.7.19.124:33003
I20260928 12:26:26.093734 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.080517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:59405 (local address 127.7.19.125:33567)
0928 12:26:26.080561 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.080563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.080853 (+ 290us) server_negotiation.cc:407] Connection header received
0928 12:26:26.082098 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.082100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.082113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.082168 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.082917 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.083537 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.084557 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.084869 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.084937 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.088105 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.088107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.088107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.088115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.091990 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.092596 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.092598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.092598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.092655 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.093018 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.093019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.093020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.093154 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.093236 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.093511 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.093623 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:26.093763 14582 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:26.093950 14582 heartbeater.cc:511] Master 127.7.19.124:33003 requested a full tablet report, sending...
I20260928 12:26:26.094380 14584 heartbeater.cc:347] Connected to a master server at 127.7.19.125:33567
I20260928 12:26:26.094391 14130 ts_manager.cc:194] Registered new tserver with Master: ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749)
I20260928 12:26:26.094426 14584 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:26.094493 14584 heartbeater.cc:511] Master 127.7.19.125:33567 requested a full tablet report, sending...
I20260928 12:26:26.094659 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.080519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:57217 (local address 127.7.19.126:45679)
0928 12:26:26.080972 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.080974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.080984 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:26.082019 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.082020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.082033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.082091 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.082920 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.083648 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.084701 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.084995 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.085082 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.089074 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.089076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.089076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.089084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.090976 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.093862 (+ 2886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.093863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.093863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.093920 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.094110 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.094110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.094111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.094243 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.094320 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.094515 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.094592 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421}
I20260928 12:26:26.095017 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.080529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:57217)
0928 12:26:26.080713 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.080753 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.081803 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.082178 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.082182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.082592 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:26.082812 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.082818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.083737 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.083738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.084601 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.084603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.085082 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.085083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.085162 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.086925 (+ 1763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:26.086930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:26.088972 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:26.093261 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:26.093262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:26.093262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:26.093805 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:26.093995 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:26.093995 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:26.093996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:26.094056 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:26.094354 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:26.094355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:26.094399 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.094777 (+ 378us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.094878 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":34,"thread_start_us":81,"threads_started":1}
I20260928 12:26:26.095033 13944 ts_manager.cc:194] Registered new tserver with Master: ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749)
I20260928 12:26:26.095439 14583 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45679
I20260928 12:26:26.095496 14583 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:26.095570 14583 heartbeater.cc:511] Master 127.7.19.126:45679 requested a full tablet report, sending...
I20260928 12:26:26.095793 13347 ts_manager.cc:194] Re-registered known tserver with Master: ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749)
I20260928 12:26:26.096321 13347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:57217
I20260928 12:26:26.097180 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.087138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.7.19.125:33567)
0928 12:26:26.088897 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.088899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.088906 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:26:26.088928 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.088928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.088979 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.089963 (+ 984us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.089997 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.090712 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.091671 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.091992 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.092053 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.093480 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.093481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.093482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.093488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.095783 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.096136 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.096137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.096138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.096204 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.096428 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.096429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.096429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.096571 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.096689 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.097021 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.097097 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":52,"threads_started":1}
I20260928 12:26:26.098852 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.087176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.7.19.126:45679)
0928 12:26:26.087448 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.087450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.087458 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:26.087577 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.087579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.087594 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.087671 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.089083 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.089887 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.090939 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.091235 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.091290 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.095636 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.095637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.095638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.095645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.097735 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.098100 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.098101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.098101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.098150 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.098388 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.098389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.098389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.098482 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.098551 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.098671 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.098755 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":72,"threads_started":1}
I20260928 12:26:26.100421 13347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:26.102828 13347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:26.103410 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.088409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.7.19.124:33003)
0928 12:26:26.091729 (+ 3320us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.091731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.091738 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:26:26.091756 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.091756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.091766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.091816 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.097039 (+ 5223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.097809 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.098768 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.099065 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.099147 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.100706 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.100707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.100707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.100714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.102365 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.102682 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.102683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.102684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.102743 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.102960 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.102961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.102961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.103076 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.103164 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.103265 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.103341 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3304,"thread_start_us":71,"threads_started":1}
I20260928 12:26:26.110263 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.106518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:40657 (local address 127.0.0.1:39016)
0928 12:26:26.106565 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.106575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.106610 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.107119 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.107120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.107121 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:26.107156 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.107157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.108197 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.108198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.109188 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.109190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.109995 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.109996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.110062 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.110067 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.110149 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.110193 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:26:26.110544 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.106769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.7.19.65:40657)
0928 12:26:26.106975 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.106980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.106990 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:26.107029 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.107032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.107049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.107122 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:26.107240 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.108091 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.109297 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.109888 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.109993 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.110075 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.110165 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.110253 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260928 12:26:26.111116 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.106518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:33749 (local address 127.0.0.1:45128)
0928 12:26:26.106793 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.106803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.106849 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.107401 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.107403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.107405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:26.107465 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.107466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.108772 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.108773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.109642 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.109644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.110808 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.110809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.110886 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.110891 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.110993 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.111035 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":170,"thread_start_us":86,"threads_started":1}
I20260928 12:26:26.111275 14243 tablet_service.cc:1467] Processing CreateTablet for tablet 51ea4243abfd402eb11def75cf888328 (DEFAULT_TABLE table=test-table [id=33c2c1104c274346816238b24304ce60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:26.111526 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.106614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.7.19.67:33749)
0928 12:26:26.107237 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.107242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.107252 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:26.107284 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.107286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.107303 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.107385 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:26.107549 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.108595 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.109743 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.110547 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.110621 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.110684 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.111013 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.111416 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":86,"threads_started":1}
I20260928 12:26:26.111642 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ea4243abfd402eb11def75cf888328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:26.112099 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.106593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39469 (local address 127.0.0.1:54802)
0928 12:26:26.106741 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.106752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.106817 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.107499 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.107500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.107502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:26.107560 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.107561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.108853 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.108854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.110785 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.110788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.111872 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.111873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.111931 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.111935 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.111993 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.112037 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":103,"threads_started":1}
I20260928 12:26:26.112366 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.107019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.7.19.66:39469)
0928 12:26:26.107291 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.107295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.107304 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:26.107338 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.107340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.107354 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.107421 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:26.107639 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.108731 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.110894 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.111710 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.111774 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.111839 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.112033 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.112065 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":79,"threads_started":1}
I20260928 12:26:26.112980 14380 tablet_service.cc:1467] Processing CreateTablet for tablet 51ea4243abfd402eb11def75cf888328 (DEFAULT_TABLE table=test-table [id=33c2c1104c274346816238b24304ce60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:26.113281 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ea4243abfd402eb11def75cf888328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:26.114600 14608 tablet_bootstrap.cc:492] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Bootstrap starting.
I20260928 12:26:26.115289 14608 tablet_bootstrap.cc:654] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:26.115199 14516 tablet_service.cc:1467] Processing CreateTablet for tablet 51ea4243abfd402eb11def75cf888328 (DEFAULT_TABLE table=test-table [id=33c2c1104c274346816238b24304ce60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:26.115510 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ea4243abfd402eb11def75cf888328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:26.115621 14608 log.cc:824] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:26.116353 14610 tablet_bootstrap.cc:492] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Bootstrap starting.
I20260928 12:26:26.116382 14608 tablet_bootstrap.cc:492] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: No bootstrap required, opened a new log
I20260928 12:26:26.116468 14608 ts_tablet_manager.cc:1397] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:26.117444 14610 tablet_bootstrap.cc:654] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:26.117728 14610 log.cc:824] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:26.118273 14608 raft_consensus.cc:348] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.118458 14608 raft_consensus.cc:374] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:26.118467 14610 tablet_bootstrap.cc:492] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: No bootstrap required, opened a new log
I20260928 12:26:26.118502 14608 raft_consensus.cc:729] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a732fe7ce34d5f87f84824c9c012b1, State: Initialized, Role: FOLLOWER
I20260928 12:26:26.118523 14610 ts_tablet_manager.cc:1397] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:26:26.118610 14608 consensus_queue.cc:260] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.118865 14608 ts_tablet_manager.cc:1428] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:26.119023 14310 heartbeater.cc:503] Master 127.7.19.126:45679 was elected leader, sending a full tablet report...
I20260928 12:26:26.120250 14610 raft_consensus.cc:348] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.120362 14610 raft_consensus.cc:374] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:26.120384 14610 raft_consensus.cc:729] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750c8f17c9d749d794aeb3e6fd4a079f, State: Initialized, Role: FOLLOWER
I20260928 12:26:26.120494 14610 consensus_queue.cc:260] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.120756 14610 ts_tablet_manager.cc:1428] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:26:26.121047 14447 heartbeater.cc:503] Master 127.7.19.126:45679 was elected leader, sending a full tablet report...
I20260928 12:26:26.121218 14613 tablet_bootstrap.cc:492] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Bootstrap starting.
I20260928 12:26:26.121881 14613 tablet_bootstrap.cc:654] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:26.122239 14613 log.cc:824] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:26.122830 14613 tablet_bootstrap.cc:492] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: No bootstrap required, opened a new log
I20260928 12:26:26.122877 14613 ts_tablet_manager.cc:1397] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:26.124769 14613 raft_consensus.cc:348] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.124887 14613 raft_consensus.cc:374] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:26.124907 14613 raft_consensus.cc:729] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb72fb9528747f59e66de91130dedc9, State: Initialized, Role: FOLLOWER
I20260928 12:26:26.125034 14613 consensus_queue.cc:260] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.125250 14613 ts_tablet_manager.cc:1428] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:26:26.125326 14583 heartbeater.cc:503] Master 127.7.19.126:45679 was elected leader, sending a full tablet report...
I20260928 12:26:26.164342 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.161959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37802)
0928 12:26:26.162015 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.162027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.162079 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.162228 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.162229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.162231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:26.162284 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.162286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.162962 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.162962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.163652 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.163653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.164171 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.164171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.164212 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.164215 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.164254 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.164289 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:26:26.164341 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.162033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.7.19.124:33003)
0928 12:26:26.162092 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.162094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.162103 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:26.162132 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.162133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.162150 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.162206 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:26.162378 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.162895 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.163723 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.164102 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.164159 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.164211 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.164256 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.164281 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:26.164795 14140 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 12:26:26.165789 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c93fe733b14ced97fdb4d141471377. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:26.165901 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: This master's current role is: LEARNER
W20260928 12:26:26.202560 14449 tablet.cc:2369] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:26:26.313975 14312 tablet.cc:2369] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:26:26.330559 14585 tablet.cc:2369] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:26.382061 14616 raft_consensus.cc:482] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:26.382207 14616 raft_consensus.cc:504] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.382588 14616 leader_election.cc:288] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657), 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:26.386631 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.382832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39469 (local address 127.7.19.67:57829)
0928 12:26:26.382899 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.382912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.382957 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.383120 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.383121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.383122 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:26.383221 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.383223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.384167 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.384168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.385476 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.385479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.386347 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.386348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.386425 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.386430 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.386495 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.386536 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":21}
I20260928 12:26:26.386639 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.382908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:57829 (local address 127.7.19.66:39469)
0928 12:26:26.382955 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.382957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.382970 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:26:26.383008 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.383009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.383022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.383088 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:26.383317 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.384083 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.385577 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.386243 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.386348 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.386360 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.386523 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.386569 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:26.386663 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.382907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:44841 (local address 127.7.19.65:40657)
0928 12:26:26.382968 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.382970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.382979 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:26.383035 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.383036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.383046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.383095 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:26.383316 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.384067 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.385607 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.386269 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.386348 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.386360 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.386546 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.386586 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:26:26.386637 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.382832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:40657 (local address 127.7.19.67:44841)
0928 12:26:26.382899 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:26.382912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:26.382957 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:26.383120 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:26.383121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:26.383122 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:26.383221 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.383223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.384171 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.384172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.385476 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:26.385479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.386377 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.386378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:26.386443 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.386446 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:26.386498 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:26:26.386536 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":166,"spinlock_wait_cycles":768}
I20260928 12:26:26.387074 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ea4243abfd402eb11def75cf888328" candidate_uuid: "ddb72fb9528747f59e66de91130dedc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" is_pre_election: true
I20260928 12:26:26.387259 14400 raft_consensus.cc:2454] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb72fb9528747f59e66de91130dedc9 in term 0.
I20260928 12:26:26.387240 14263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ea4243abfd402eb11def75cf888328" candidate_uuid: "ddb72fb9528747f59e66de91130dedc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a732fe7ce34d5f87f84824c9c012b1" is_pre_election: true
I20260928 12:26:26.387400 14263 raft_consensus.cc:2454] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb72fb9528747f59e66de91130dedc9 in term 0.
I20260928 12:26:26.387516 14469 leader_election.cc:302] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750c8f17c9d749d794aeb3e6fd4a079f, ddb72fb9528747f59e66de91130dedc9; no voters:
I20260928 12:26:26.387712 14616 raft_consensus.cc:2781] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:26.387776 14616 raft_consensus.cc:482] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:26.387794 14616 raft_consensus.cc:3037] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:26.388520 14616 raft_consensus.cc:504] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.388659 14616 leader_election.cc:288] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [CANDIDATE]: Term 1 election: Requested vote from peers 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657), 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:26.388827 14263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ea4243abfd402eb11def75cf888328" candidate_uuid: "ddb72fb9528747f59e66de91130dedc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a732fe7ce34d5f87f84824c9c012b1"
I20260928 12:26:26.388904 14263 raft_consensus.cc:3037] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:26.388861 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ea4243abfd402eb11def75cf888328" candidate_uuid: "ddb72fb9528747f59e66de91130dedc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750c8f17c9d749d794aeb3e6fd4a079f"
I20260928 12:26:26.388983 14400 raft_consensus.cc:3037] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:26.389806 14263 raft_consensus.cc:2454] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb72fb9528747f59e66de91130dedc9 in term 1.
I20260928 12:26:26.389997 14471 leader_election.cc:302] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a732fe7ce34d5f87f84824c9c012b1, ddb72fb9528747f59e66de91130dedc9; no voters:
I20260928 12:26:26.390022 14400 raft_consensus.cc:2454] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb72fb9528747f59e66de91130dedc9 in term 1.
I20260928 12:26:26.390097 14616 raft_consensus.cc:2781] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:26.390251 14616 raft_consensus.cc:686] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 1 LEADER]: Becoming Leader. State: Replica: ddb72fb9528747f59e66de91130dedc9, State: Running, Role: LEADER
I20260928 12:26:26.390384 14616 consensus_queue.cc:237] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } }
I20260928 12:26:26.391125 13347 catalog_manager.cc:5495] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb72fb9528747f59e66de91130dedc9 (127.7.19.67). New cstate: current_term: 1 leader_uuid: "ddb72fb9528747f59e66de91130dedc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:26.391810 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: NON_VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: true } } }
I20260928 12:26:26.391922 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:26.392542 13905 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: attempting to promote NON_VOTER 46ceaa3e4b3a4897a221c1a5e2898179 to VOTER
I20260928 12:26:26.392652 13905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } }
I20260928 12:26:26.393198 13954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Refusing update from remote peer 61c93fe733b14ced97fdb4d141471377: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 12:26:26.393232 14140 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 LEARNER]: Refusing update from remote peer 61c93fe733b14ced97fdb4d141471377: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 12:26:26.393419 14094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 12:26:26.393500 14623 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 12:26:26.394133 14624 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.394289 13954 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.394279 14140 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.395130 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.395222 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:26:26.395179 14624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.395351 14624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:26.395380 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.395372 14626 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.395454 14626 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:26:26.395396 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c93fe733b14ced97fdb4d141471377" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:26.395502 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:26.395464 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:26.464013 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.461689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.7.19.67:33749)
0928 12:26:26.461744 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.461746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.461755 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:26.461803 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.461804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.461823 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.461894 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:26.462017 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.462587 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.463314 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.463536 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.463588 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.463732 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:26.463828 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:26.463894 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.463927 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.463944 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:26:26.465804 14400 raft_consensus.cc:1261] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Refusing update from remote peer ddb72fb9528747f59e66de91130dedc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:26.465804 14263 raft_consensus.cc:1261] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 FOLLOWER]: Refusing update from remote peer ddb72fb9528747f59e66de91130dedc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:26.466185 14616 consensus_queue.cc:1035] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:26.466333 14621 consensus_queue.cc:1035] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:26.468510 14628 mvcc.cc:205] Tried to move back new op lower bound from 7334290990961266688 to 7334290990655488000. Current Snapshot: MvccSnapshot[applied={T|T < 7334290990961266688}]
I20260928 12:26:26.469713 14629 mvcc.cc:205] Tried to move back new op lower bound from 7334290990961266688 to 7334290990655488000. Current Snapshot: MvccSnapshot[applied={T|T < 7334290990961266688 or (T in {7334290990961266688})}]
I20260928 12:26:26.472724 13347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:26.472828 13347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:26.474157 13347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61c93fe733b14ced97fdb4d141471377: Sending DeleteTablet for 3 replicas of tablet 51ea4243abfd402eb11def75cf888328
I20260928 12:26:26.474493 14516 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ea4243abfd402eb11def75cf888328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:26 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:45128
I20260928 12:26:26.474643 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ea4243abfd402eb11def75cf888328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:26 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:54802
I20260928 12:26:26.474757 14638 tablet_replica.cc:330] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: stopping tablet replica
I20260928 12:26:26.474824 14639 tablet_replica.cc:330] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: stopping tablet replica
I20260928 12:26:26.474846 14638 raft_consensus.cc:2229] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:26.474915 14639 raft_consensus.cc:2229] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:26.474962 14638 raft_consensus.cc:2258] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:26.475006 14639 raft_consensus.cc:2258] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:26.475386 14638 ts_tablet_manager.cc:1905] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:26.475883 14243 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ea4243abfd402eb11def75cf888328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:26 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:39016
I20260928 12:26:26.476058 14641 tablet_replica.cc:330] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: stopping tablet replica
I20260928 12:26:26.476155 14641 raft_consensus.cc:2229] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:26.476261 14641 raft_consensus.cc:2258] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:26.476567 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13315
I20260928 12:26:26.476708 14641 ts_tablet_manager.cc:1905] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:26.477669 14638 ts_tablet_manager.cc:1918] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:26.477731 14638 log.cc:1196] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/51ea4243abfd402eb11def75cf888328
I20260928 12:26:26.478000 14638 ts_tablet_manager.cc:1939] T 51ea4243abfd402eb11def75cf888328 P ddb72fb9528747f59e66de91130dedc9: Deleting consensus metadata
I20260928 12:26:26.478209 14641 ts_tablet_manager.cc:1918] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:26.478271 14641 log.cc:1196] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/51ea4243abfd402eb11def75cf888328
I20260928 12:26:26.478530 14641 ts_tablet_manager.cc:1939] T 51ea4243abfd402eb11def75cf888328 P 19a732fe7ce34d5f87f84824c9c012b1: Deleting consensus metadata
I20260928 12:26:26.481182 14639 ts_tablet_manager.cc:1905] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:26.482756 14639 ts_tablet_manager.cc:1918] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:26.482833 14639 log.cc:1196] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/51ea4243abfd402eb11def75cf888328
I20260928 12:26:26.483080 14639 ts_tablet_manager.cc:1939] T 51ea4243abfd402eb11def75cf888328 P 750c8f17c9d749d794aeb3e6fd4a079f: Deleting consensus metadata
W20260928 12:26:26.483606 14336 connection.cc:535] server connection from 127.0.0.1:54802 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:26:26.483889 14469 connection.cc:535] server connection from 127.0.0.1:45128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 12:26:26.484249 14199 connection.cc:535] server connection from 127.0.0.1:39016 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:26:26.489277 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.475317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.7.19.124:33003)
0928 12:26:26.475356 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.475358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.480896 (+ 5538us) server_negotiation.cc:407] Connection header received
0928 12:26:26.480956 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.480957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.480969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.481026 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.481831 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.482650 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.483778 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.484086 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.484155 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.486120 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.486122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.486122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.486129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.487893 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.488264 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.488265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.488265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.488314 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.488509 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.488510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.488510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.488627 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.489072 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.489112 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.489183 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:26:26.493431 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.485411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.7.19.125:33567)
0928 12:26:26.485455 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.485457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.485561 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:26:26.485674 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.485675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.485688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.485722 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.486510 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.487267 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.488332 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.488641 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.488719 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.490525 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.490526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.490527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.490534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.492182 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.492626 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.492627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.492627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.492702 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.492915 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.492916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.492917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.493041 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.493150 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.493240 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.493334 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:26:26.493537 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:26.485614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.7.19.124:33003)
0928 12:26:26.485650 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:26.485652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:26.485838 (+ 186us) server_negotiation.cc:407] Connection header received
0928 12:26:26.485968 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:26.485986 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:26.485998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:26.486028 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:26.486715 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.487395 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.488332 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:26.488615 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:26.488652 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:26.490566 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:26.490567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:26.490567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:26.490573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:26.492169 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.492589 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.492590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.492590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.492647 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:26.492953 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:26.492954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:26.492954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:26.493074 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:26.493162 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:26.493341 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 12:26:26.493412 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:26:27.470155 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.469869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:43837)
0928 12:26:27.470021 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.470099 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260928 12:26:27.470373 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.470136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:39123)
0928 12:26:27.470253 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.470319 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":49,"threads_started":1}
W20260928 12:26:27.470620 14447 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
W20260928 12:26:27.470643 14310 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
I20260928 12:26:27.471469 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.471225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:35243)
0928 12:26:27.471368 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.471429 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
W20260928 12:26:27.471735 14583 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:45679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
I20260928 12:26:27.974496 14651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61c93fe733b14ced97fdb4d141471377)
I20260928 12:26:27.974653 14651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } }
I20260928 12:26:27.974761 14652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61c93fe733b14ced97fdb4d141471377)
I20260928 12:26:27.974920 14652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } }
I20260928 12:26:27.975169 14651 leader_election.cc:288] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679), 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124:33003)
I20260928 12:26:27.975409 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.975066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39008)
0928 12:26:27.975301 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.975363 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":49,"threads_started":1}
I20260928 12:26:27.975421 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.975253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39010)
0928 12:26:27.975332 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.975376 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":46,"threads_started":1}
W20260928 12:26:27.975534 14115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
W20260928 12:26:27.975591 13929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
I20260928 12:26:27.975598 14652 leader_election.cc:288] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679), a5838c5c62e34597bf56643d85bcb27a (127.7.19.125:33567)
I20260928 12:26:27.976046 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.975892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39022)
0928 12:26:27.975990 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.976018 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260928 12:26:27.976157 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.976081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39038)
0928 12:26:27.976114 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.976134 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:26:27.976197 14115 leader_election.cc:334] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679): Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
W20260928 12:26:27.976327 13929 leader_election.cc:334] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679): Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
I20260928 12:26:27.978300 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.975093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.7.19.124:33003)
0928 12:26:27.975277 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:27.975279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:27.975287 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:27.975310 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:27.975310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:27.975321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:27.975539 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:27.975574 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.976335 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.977546 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.977957 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.978048 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:27.978060 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:27.978201 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:26:27.978238 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":165,"thread_start_us":55,"threads_started":1}
I20260928 12:26:27.978313 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.974993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:33003 (local address 127.0.0.1:37836)
0928 12:26:27.975110 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.975122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:27.975174 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:27.975405 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:27.975406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:27.975408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:27.975471 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:27.975473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.976454 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.976455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:27.977461 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:27.977463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.978051 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.978052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:27.978118 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:27.978121 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:27.978194 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:26:27.978241 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":49,"threads_started":1}
I20260928 12:26:27.978564 14140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5838c5c62e34597bf56643d85bcb27a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" is_pre_election: true
I20260928 12:26:27.978688 14140 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5838c5c62e34597bf56643d85bcb27a in term 1.
I20260928 12:26:27.978754 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.975536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.7.19.125:33567)
0928 12:26:27.975686 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:27.975688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:27.975897 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:26:27.975966 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:27.975967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:27.975979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:27.976023 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:27.976209 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.976976 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.978007 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.978404 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.978485 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:27.978496 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:27.978654 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 12:26:27.978690 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":45,"threads_started":1}
I20260928 12:26:27.978814 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.975432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:33567 (local address 127.0.0.1:57738)
0928 12:26:27.975815 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.975824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:27.975888 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:27.976089 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:27.976089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:27.976091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:27.976135 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:27.976136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.977047 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.977048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:27.977907 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:27.977909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:27.978537 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:27.978538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:27.978579 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:27.978582 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:27.978649 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 12:26:27.978688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":19,"thread_start_us":35,"threads_started":1}
I20260928 12:26:27.978869 13930 leader_election.cc:302] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46ceaa3e4b3a4897a221c1a5e2898179, a5838c5c62e34597bf56643d85bcb27a; no voters: 61c93fe733b14ced97fdb4d141471377
I20260928 12:26:27.978977 14651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 12:26:27.979043 14651 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61c93fe733b14ced97fdb4d141471377)
I20260928 12:26:27.979079 14651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:26:27.979059 13954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5838c5c62e34597bf56643d85bcb27a" is_pre_election: true
I20260928 12:26:27.979738 14651 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } }
I20260928 12:26:27.979866 13954 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46ceaa3e4b3a4897a221c1a5e2898179 in current term 2: Already voted for candidate a5838c5c62e34597bf56643d85bcb27a in this term.
I20260928 12:26:27.979880 14651 leader_election.cc:288] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 2 election: Requested vote from peers 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679), 46ceaa3e4b3a4897a221c1a5e2898179 (127.7.19.124:33003)
I20260928 12:26:27.980011 14140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5838c5c62e34597bf56643d85bcb27a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46ceaa3e4b3a4897a221c1a5e2898179"
I20260928 12:26:27.980059 14116 leader_election.cc:302] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46ceaa3e4b3a4897a221c1a5e2898179; no voters: 61c93fe733b14ced97fdb4d141471377, a5838c5c62e34597bf56643d85bcb27a
I20260928 12:26:27.980087 14140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:26:27.980090 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.979947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39042)
0928 12:26:27.980040 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.980066 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 12:26:27.980765 14140 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5838c5c62e34597bf56643d85bcb27a in term 2.
I20260928 12:26:27.980863 14652 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:26:27.980934 13930 leader_election.cc:302] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46ceaa3e4b3a4897a221c1a5e2898179, a5838c5c62e34597bf56643d85bcb27a; no voters:
I20260928 12:26:27.981006 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:27.980917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39050)
0928 12:26:27.980961 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:27.980983 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:26:27.981005 14651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 2 FOLLOWER]: Leader election won for term 2
W20260928 12:26:27.981081 13929 leader_election.cc:334] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679): Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
I20260928 12:26:27.981159 14651 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [term 2 LEADER]: Becoming Leader. State: Replica: a5838c5c62e34597bf56643d85bcb27a, State: Running, Role: LEADER
I20260928 12:26:27.981251 14651 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } }
I20260928 12:26:27.981578 14662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5838c5c62e34597bf56643d85bcb27a. Latest consensus state: current_term: 2 leader_uuid: "a5838c5c62e34597bf56643d85bcb27a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:27.981680 14662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:27.981855 14664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:27.982218 14664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:27.982270 14664 catalog_manager.cc:1259] Loaded cluster ID: f8b2201c43f54977a0ebb4962f589b2f
I20260928 12:26:27.982286 14664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:27.982407 14664 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:27.982461 14664 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Loaded TSK: 0
I20260928 12:26:27.982549 14664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:28.031034 13943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:28.032619 13943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:28.033659 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.033550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39064)
0928 12:26:28.033601 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.033629 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:26:28.033690 14140 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [term 2 FOLLOWER]: Refusing update from remote peer a5838c5c62e34597bf56643d85bcb27a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 12:26:28.033901 14662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [LEADER]: Connected to new peer: Peer: permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 12:26:28.034026 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.033921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.0.0.1:39068)
0928 12:26:28.033976 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.034003 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 12:26:28.034191 13929 consensus_peers.cc:487] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a -> Peer 61c93fe733b14ced97fdb4d141471377 (127.7.19.126:45679): Couldn't send request to peer 61c93fe733b14ced97fdb4d141471377. Status: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 12:26:28.034482 14652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5838c5c62e34597bf56643d85bcb27a. Latest consensus state: current_term: 2 leader_uuid: "a5838c5c62e34597bf56643d85bcb27a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:28.034587 14652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:26:28.034922 14665 mvcc.cc:205] Tried to move back new op lower bound from 7334290997384069120 to 7334290997171777536. Current Snapshot: MvccSnapshot[applied={T|T < 7334290997384069120}]
I20260928 12:26:28.035038 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a5838c5c62e34597bf56643d85bcb27a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:28.035137 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:28.035126 14652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a5838c5c62e34597bf56643d85bcb27a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "61c93fe733b14ced97fdb4d141471377" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45679 } } peers { permanent_uuid: "a5838c5c62e34597bf56643d85bcb27a" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 33567 } attrs { promote: false } } peers { permanent_uuid: "46ceaa3e4b3a4897a221c1a5e2898179" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 33003 } attrs { promote: false } } }
I20260928 12:26:28.035207 14652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ceaa3e4b3a4897a221c1a5e2898179 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:26:28.035342 14666 mvcc.cc:205] Tried to move back new op lower bound from 7334290997384069120 to 7334290997171777536. Current Snapshot: MvccSnapshot[applied={T|T < 7334290997384069120 or (T in {7334290997384069120})}]
I20260928 12:26:28.039940 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.036931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:40657 (local address 127.0.0.1:39020)
0928 12:26:28.037075 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.037086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:28.037138 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:28.037415 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:28.037415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:28.037417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:28.037452 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.037453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.038383 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.038384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.039092 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.039094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.039711 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.039712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.039768 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.039772 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:28.039827 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:26:28.039867 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260928 12:26:28.040139 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.037115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.7.19.65:40657)
0928 12:26:28.037250 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:28.037251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:28.037259 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:28.037281 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:28.037282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:28.037300 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:28.037359 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:28.037528 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.038289 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.039184 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.039601 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.039666 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.039679 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:28.040029 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 12:26:28.040073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 12:26:28.040401 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.036837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:33749 (local address 127.0.0.1:45138)
0928 12:26:28.036874 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.036883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:28.036918 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:28.037387 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:28.037389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:28.037391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:28.037440 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.037441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.038364 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.038365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.039479 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.039481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.040215 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.040216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.040282 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.040286 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:28.040325 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 12:26:28.040361 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":9}
I20260928 12:26:28.040573 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.036899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39469 (local address 127.0.0.1:54808)
0928 12:26:28.036935 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.036944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:28.036985 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:28.037561 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:28.037562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:28.037565 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:28.037611 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.037613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.038268 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.038269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.039499 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.039501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.040326 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.040327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.040391 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.040394 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:28.040466 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 12:26:28.040510 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":43}
I20260928 12:26:28.040591 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.037254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.7.19.66:39469)
0928 12:26:28.037424 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:28.037427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:28.037435 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:28.037458 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:28.037459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:28.037470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:28.037677 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:28.037709 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.038172 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.039647 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.040242 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.040324 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.040337 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:28.040484 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:26:28.040526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260928 12:26:28.040614 14243 tablet_service.cc:1467] Processing CreateTablet for tablet a9ebe5c59b2449efad56d4436eeb3291 (DEFAULT_TABLE table=test-table [id=afb1303cb8b44c49b71a2b513b1231a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:28.040781 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ebe5c59b2449efad56d4436eeb3291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:28.040870 14380 tablet_service.cc:1467] Processing CreateTablet for tablet a9ebe5c59b2449efad56d4436eeb3291 (DEFAULT_TABLE table=test-table [id=afb1303cb8b44c49b71a2b513b1231a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:28.040971 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ebe5c59b2449efad56d4436eeb3291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:28.041903 14674 tablet_bootstrap.cc:492] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1: Bootstrap starting.
I20260928 12:26:28.042182 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.037094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.7.19.67:33749)
0928 12:26:28.037237 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:28.037240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:28.037249 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:28.037277 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:28.037278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:28.037289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:28.037504 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:28.037541 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.038289 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.039567 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.040119 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.042062 (+ 1943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.042084 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:28.042100 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 12:26:28.042134 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 12:26:28.042313 14674 tablet_bootstrap.cc:654] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:28.042418 14516 tablet_service.cc:1467] Processing CreateTablet for tablet a9ebe5c59b2449efad56d4436eeb3291 (DEFAULT_TABLE table=test-table [id=afb1303cb8b44c49b71a2b513b1231a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:28.042436 14675 tablet_bootstrap.cc:492] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Bootstrap starting.
I20260928 12:26:28.042541 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ebe5c59b2449efad56d4436eeb3291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:28.042760 14675 tablet_bootstrap.cc:654] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:28.043272 14675 tablet_bootstrap.cc:492] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: No bootstrap required, opened a new log
I20260928 12:26:28.043314 14675 ts_tablet_manager.cc:1397] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:28.043429 14675 raft_consensus.cc:348] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.043486 14675 raft_consensus.cc:374] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:28.043500 14675 raft_consensus.cc:729] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750c8f17c9d749d794aeb3e6fd4a079f, State: Initialized, Role: FOLLOWER
I20260928 12:26:28.043565 14675 consensus_queue.cc:260] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.043656 14678 tablet_bootstrap.cc:492] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Bootstrap starting.
I20260928 12:26:28.043731 14675 ts_tablet_manager.cc:1428] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:28.044059 14674 tablet_bootstrap.cc:492] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1: No bootstrap required, opened a new log
I20260928 12:26:28.044054 14678 tablet_bootstrap.cc:654] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:28.044107 14674 ts_tablet_manager.cc:1397] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:28.044255 14674 raft_consensus.cc:348] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.044317 14674 raft_consensus.cc:374] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:28.044335 14674 raft_consensus.cc:729] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a732fe7ce34d5f87f84824c9c012b1, State: Initialized, Role: FOLLOWER
I20260928 12:26:28.044340 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.044102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:43983)
0928 12:26:28.044259 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.044294 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260928 12:26:28.044384 14674 consensus_queue.cc:260] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.044502 14674 ts_tablet_manager.cc:1428] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:28.044776 14678 tablet_bootstrap.cc:492] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: No bootstrap required, opened a new log
I20260928 12:26:28.044827 14678 ts_tablet_manager.cc:1397] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:28.044929 14678 raft_consensus.cc:348] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.044989 14678 raft_consensus.cc:374] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:28.045011 14678 raft_consensus.cc:729] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb72fb9528747f59e66de91130dedc9, State: Initialized, Role: FOLLOWER
I20260928 12:26:28.045053 14678 consensus_queue.cc:260] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.045185 14678 ts_tablet_manager.cc:1428] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:28.045473 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.045354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:54761)
0928 12:26:28.045420 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.045448 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 12:26:28.045651 14448 heartbeater.cc:503] Master 127.7.19.125:33567 was elected leader, sending a full tablet report...
I20260928 12:26:28.045776 14584 heartbeater.cc:503] Master 127.7.19.125:33567 was elected leader, sending a full tablet report...
I20260928 12:26:28.045779 14311 heartbeater.cc:503] Master 127.7.19.125:33567 was elected leader, sending a full tablet report...
I20260928 12:26:28.046267 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.044698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:41413)
0928 12:26:28.046194 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.046232 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1475,"thread_start_us":72,"threads_started":1}
I20260928 12:26:28.046461 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.045405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:34901)
0928 12:26:28.046394 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.046426 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":89,"threads_started":1}
I20260928 12:26:28.046593 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.046496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:34195)
0928 12:26:28.046548 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.046573 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:26:28.046739 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.046671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:52935)
0928 12:26:28.046699 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.046720 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 12:26:28.133581 14682 raft_consensus.cc:482] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:28.133680 14682 raft_consensus.cc:504] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.133963 14682 leader_election.cc:288] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749), 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:28.136933 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.134236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51801 (local address 127.7.19.66:39469)
0928 12:26:28.134284 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:28.134285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:28.134484 (+ 199us) server_negotiation.cc:407] Connection header received
0928 12:26:28.134548 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:28.134550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:28.134561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:28.134639 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:28.134799 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.135248 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.136184 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.136556 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.136636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.136649 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:28.136825 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:26:28.136862 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:26:28.136951 14671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.134280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:54021 (local address 127.7.19.67:33749)
0928 12:26:28.134338 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:28.134340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:28.134349 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:28.134375 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:28.134376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:28.134388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:28.134461 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:28.134631 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.135112 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.136176 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.136560 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.136644 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.136658 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:28.136839 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 12:26:28.136876 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:28.136953 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.134143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:33749 (local address 127.7.19.65:54021)
0928 12:26:28.134209 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.134222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:28.134292 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:28.134483 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:28.134484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:28.134487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:28.134560 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.134562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.135194 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.135195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.136089 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.136091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.136684 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.136685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.136752 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.136755 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:28.136830 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:26:28.136872 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":32}
I20260928 12:26:28.136963 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.134143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39469 (local address 127.7.19.65:51801)
0928 12:26:28.134405 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.134415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:28.134491 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:28.134646 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:28.134647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:28.134648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:28.134714 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.134716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.135329 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.135330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.136090 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:28.136092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.136636 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.136637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:28.136730 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.136734 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:28.136816 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:28.136858 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":76,"thread_start_us":141,"threads_started":1}
I20260928 12:26:28.137212 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ebe5c59b2449efad56d4436eeb3291" candidate_uuid: "19a732fe7ce34d5f87f84824c9c012b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb72fb9528747f59e66de91130dedc9" is_pre_election: true
I20260928 12:26:28.137356 14536 raft_consensus.cc:2454] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a732fe7ce34d5f87f84824c9c012b1 in term 0.
I20260928 12:26:28.137434 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ebe5c59b2449efad56d4436eeb3291" candidate_uuid: "19a732fe7ce34d5f87f84824c9c012b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" is_pre_election: true
I20260928 12:26:28.137559 14400 raft_consensus.cc:2454] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19a732fe7ce34d5f87f84824c9c012b1 in term 0.
I20260928 12:26:28.137552 14197 leader_election.cc:302] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a732fe7ce34d5f87f84824c9c012b1, ddb72fb9528747f59e66de91130dedc9; no voters:
I20260928 12:26:28.137774 14682 raft_consensus.cc:2781] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:28.137813 14682 raft_consensus.cc:482] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:28.137827 14682 raft_consensus.cc:3037] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:28.138513 14682 raft_consensus.cc:504] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.138652 14682 leader_election.cc:288] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [CANDIDATE]: Term 1 election: Requested vote from peers ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749), 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469)
I20260928 12:26:28.138800 14536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ebe5c59b2449efad56d4436eeb3291" candidate_uuid: "19a732fe7ce34d5f87f84824c9c012b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb72fb9528747f59e66de91130dedc9"
I20260928 12:26:28.138818 14400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ebe5c59b2449efad56d4436eeb3291" candidate_uuid: "19a732fe7ce34d5f87f84824c9c012b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750c8f17c9d749d794aeb3e6fd4a079f"
I20260928 12:26:28.138871 14536 raft_consensus.cc:3037] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:28.138886 14400 raft_consensus.cc:3037] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:28.139509 14536 raft_consensus.cc:2454] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a732fe7ce34d5f87f84824c9c012b1 in term 1.
I20260928 12:26:28.139516 14400 raft_consensus.cc:2454] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a732fe7ce34d5f87f84824c9c012b1 in term 1.
I20260928 12:26:28.139676 14197 leader_election.cc:302] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a732fe7ce34d5f87f84824c9c012b1, ddb72fb9528747f59e66de91130dedc9; no voters:
I20260928 12:26:28.139755 14682 raft_consensus.cc:2781] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:28.139909 14682 raft_consensus.cc:686] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [term 1 LEADER]: Becoming Leader. State: Replica: 19a732fe7ce34d5f87f84824c9c012b1, State: Running, Role: LEADER
I20260928 12:26:28.139992 14682 consensus_queue.cc:237] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } }
I20260928 12:26:28.140202 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.140095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:35057)
0928 12:26:28.140147 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.140180 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:26:28.140868 13943 catalog_manager.cc:5495] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "19a732fe7ce34d5f87f84824c9c012b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a732fe7ce34d5f87f84824c9c012b1" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 40657 } health_report { overall_health: HEALTHY } } }
I20260928 12:26:28.155817 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.153562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.7.19.65:40657)
0928 12:26:28.153616 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:28.153618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:28.153662 (+ 44us) server_negotiation.cc:407] Connection header received
0928 12:26:28.153695 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:28.153695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:28.153712 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:28.153773 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:28.153883 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.154311 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.155163 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:28.155343 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:28.155409 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:28.155570 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:28.155661 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:28.155710 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:28.155749 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:26:28.155766 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:26:28.157071 14400 raft_consensus.cc:1261] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Refusing update from remote peer 19a732fe7ce34d5f87f84824c9c012b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:28.157258 14536 raft_consensus.cc:1261] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Refusing update from remote peer 19a732fe7ce34d5f87f84824c9c012b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:28.157352 14687 consensus_queue.cc:1035] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750c8f17c9d749d794aeb3e6fd4a079f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:28.157547 14682 consensus_queue.cc:1035] T a9ebe5c59b2449efad56d4436eeb3291 P 19a732fe7ce34d5f87f84824c9c012b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb72fb9528747f59e66de91130dedc9" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:28.158169 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.158012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:33081)
0928 12:26:28.158087 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.158132 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 12:26:28.158654 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.158556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:33689)
0928 12:26:28.158592 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.158620 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 12:26:28.159942 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.159730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:51181)
0928 12:26:28.159877 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.159907 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260928 12:26:28.160072 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.159838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.66:44817)
0928 12:26:28.160020 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.160046 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260928 12:26:28.160794 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.160075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.67:39059)
0928 12:26:28.160734 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.160761 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":645}
I20260928 12:26:28.160950 14691 mvcc.cc:205] Tried to move back new op lower bound from 7334290997888966656 to 7334290997822332928. Current Snapshot: MvccSnapshot[applied={T|T < 7334290997888966656 or (T in {7334290997888966656})}]
I20260928 12:26:28.161005 14683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.160859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45679 (local address 127.7.19.65:34851)
0928 12:26:28.160949 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.160980 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 12:26:28.162554 13944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:28.162642 13944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:28.163941 13944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5838c5c62e34597bf56643d85bcb27a: Sending DeleteTablet for 3 replicas of tablet a9ebe5c59b2449efad56d4436eeb3291
I20260928 12:26:28.164280 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ebe5c59b2449efad56d4436eeb3291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:28 UTC) from {username='oryx', principal='oryx/127.7.19.125@KRBTEST.COM'} at 127.0.0.1:54808
I20260928 12:26:28.164304 14516 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ebe5c59b2449efad56d4436eeb3291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:28 UTC) from {username='oryx', principal='oryx/127.7.19.125@KRBTEST.COM'} at 127.0.0.1:45138
I20260928 12:26:28.164490 14243 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ebe5c59b2449efad56d4436eeb3291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:28 UTC) from {username='oryx', principal='oryx/127.7.19.125@KRBTEST.COM'} at 127.0.0.1:39020
I20260928 12:26:28.164533 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 14091
I20260928 12:26:28.165194 14702 tablet_replica.cc:330] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: stopping tablet replica
I20260928 12:26:28.165251 14702 raft_consensus.cc:2229] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:28.165297 14702 raft_consensus.cc:2258] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:28.165411 14701 tablet_replica.cc:330] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: stopping tablet replica
I20260928 12:26:28.165458 14701 raft_consensus.cc:2229] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:28.165503 14701 raft_consensus.cc:2258] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:28.165607 14702 ts_tablet_manager.cc:1905] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:28.165776 14701 ts_tablet_manager.cc:1905] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:28.167222 14702 ts_tablet_manager.cc:1918] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:28.167271 14702 log.cc:1196] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/a9ebe5c59b2449efad56d4436eeb3291
I20260928 12:26:28.167477 14702 ts_tablet_manager.cc:1939] T a9ebe5c59b2449efad56d4436eeb3291 P ddb72fb9528747f59e66de91130dedc9: Deleting consensus metadata
I20260928 12:26:28.167656 14701 ts_tablet_manager.cc:1918] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:28.167719 14701 log.cc:1196] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/a9ebe5c59b2449efad56d4436eeb3291
I20260928 12:26:28.167764 13929 catalog_manager.cc:4841] TS ddb72fb9528747f59e66de91130dedc9 (127.7.19.67:33749): tablet a9ebe5c59b2449efad56d4436eeb3291 (table test-table [id=afb1303cb8b44c49b71a2b513b1231a0]) successfully deleted
I20260928 12:26:28.167956 14701 ts_tablet_manager.cc:1939] T a9ebe5c59b2449efad56d4436eeb3291 P 750c8f17c9d749d794aeb3e6fd4a079f: Deleting consensus metadata
I20260928 12:26:28.168311 13928 catalog_manager.cc:4841] TS 750c8f17c9d749d794aeb3e6fd4a079f (127.7.19.66:39469): tablet a9ebe5c59b2449efad56d4436eeb3291 (table test-table [id=afb1303cb8b44c49b71a2b513b1231a0]) successfully deleted
I20260928 12:26:28.172415 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:28.172275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:40657 (local address 127.0.0.1:39044)
0928 12:26:28.172318 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:28.172341 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:26:28.172518 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 14317
W20260928 12:26:28.172613 13930 catalog_manager.cc:4584] TS 19a732fe7ce34d5f87f84824c9c012b1 (127.7.19.65:40657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9ebe5c59b2449efad56d4436eeb3291: Network error: Client connection negotiation failed: client connection to 127.7.19.65:40657: connect: Connection refused (error 111)
I20260928 12:26:28.177620 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 14453
I20260928 12:26:28.183733 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13811
I20260928 12:26:28.188489 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 13998
2026-09-28T12:26:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4870 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:28 dist-test-slave-cb7x krb5kdc[14706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:28 dist-test-slave-cb7x krb5kdc[14706](info): set up 2 sockets
Sep 28 12:26:28 dist-test-slave-cb7x krb5kdc[14706](info): commencing operation
krb5kdc: starting...
W20260928 12:26:29.261202 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab.
Entry for principal oryx/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab.
I20260928 12:26:29.377667 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:35971
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab
--principal=oryx/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:29.457794 14719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:29.457986 14719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:29.458007 14719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:29.459106 14719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:29.459144 14719 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 12:26:29.459157 14719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:29.459170 14719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:29.459182 14719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:29.459194 14719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:29.460588 14719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:35971
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.126.keytab
--principal=oryx/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:29.460841 14719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:29.461051 14719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:29.465685 14727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.465695 14725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:29.465756 14719 server_base.cc:1034] running on GCE node
W20260928 12:26:29.465698 14724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:29.466167 14719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:29.466454 14719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:29.467614 14719 hybrid_clock.cc:648] HybridClock initialized: now 1790598389467586 us; error 43 us; skew 500 ppm
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:29.476931 14719 init.cc:377] Logged in from keytab as oryx/127.7.19.126@KRBTEST.COM (short username oryx)
I20260928 12:26:29.477368 14719 webserver.cc:460] Webserver started at http://127.7.19.126:38127/ using document root <none> and password file <none>
I20260928 12:26:29.477548 14719 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:29.477588 14719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:29.477674 14719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:29.478627 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "a5cd341439424d5a88f3fa1059998273"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "1fc4863488508df184e8b6c385c84a4e"
server_key_iv: "c5ac995f2fd4eefb0cae779b9e3787f7"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.478909 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "a5cd341439424d5a88f3fa1059998273"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "1fc4863488508df184e8b6c385c84a4e"
server_key_iv: "c5ac995f2fd4eefb0cae779b9e3787f7"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.480110 14719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:26:29.480839 14734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:29.481040 14719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:29.481155 14719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "a5cd341439424d5a88f3fa1059998273"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "1fc4863488508df184e8b6c385c84a4e"
server_key_iv: "c5ac995f2fd4eefb0cae779b9e3787f7"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.481225 14719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:29.487496 14719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:29.489280 14719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:29.489437 14719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:29.494339 14719 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:35971
I20260928 12:26:29.494382 14796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:35971 every 8 connection(s)
I20260928 12:26:29.494695 14719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:29.495272 14797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:29.497617 14797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Bootstrap starting.
I20260928 12:26:29.498198 14797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:29.498481 14797 log.cc:824] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:29.499150 14797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: No bootstrap required, opened a new log
I20260928 12:26:29.501240 14797 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cd341439424d5a88f3fa1059998273" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35971 } }
I20260928 12:26:29.501368 14797 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:29.501390 14797 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cd341439424d5a88f3fa1059998273, State: Initialized, Role: FOLLOWER
I20260928 12:26:29.501546 14797 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cd341439424d5a88f3fa1059998273" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35971 } }
I20260928 12:26:29.501608 14797 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:29.501628 14797 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:29.501693 14797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:29.502118 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 14719
I20260928 12:26:29.502236 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:29.502388 14797 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cd341439424d5a88f3fa1059998273" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35971 } }
I20260928 12:26:29.502460 7245 external_mini_cluster.cc:1428] Setting key 35eeac1ea27aa7dbaec29ce9afe26064
I20260928 12:26:29.502496 14797 leader_election.cc:302] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5cd341439424d5a88f3fa1059998273; no voters:
I20260928 12:26:29.502656 14797 leader_election.cc:288] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:29.502699 14800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:29.502893 14800 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [term 1 LEADER]: Becoming Leader. State: Replica: a5cd341439424d5a88f3fa1059998273, State: Running, Role: LEADER
I20260928 12:26:29.502912 14797 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:29.503646 14800 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cd341439424d5a88f3fa1059998273" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35971 } }
I20260928 12:26:29.504236 14801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5cd341439424d5a88f3fa1059998273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cd341439424d5a88f3fa1059998273" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35971 } } }
I20260928 12:26:29.504310 14801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:29.504621 14802 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5cd341439424d5a88f3fa1059998273. Latest consensus state: current_term: 1 leader_uuid: "a5cd341439424d5a88f3fa1059998273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5cd341439424d5a88f3fa1059998273" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35971 } } }
I20260928 12:26:29.504727 14802 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273 [sys.catalog]: This master's current role is: LEADER
W20260928 12:26:29.504978 14816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:26:29.505033 14816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:29.505084 14818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:29.505247 14818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:29.506609 14818 catalog_manager.cc:1347] Generated new cluster ID: 8b45389c249d4263afd992d566edf9be
I20260928 12:26:29.506659 14818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:29.514070 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.503090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.7.19.126:35971)
0928 12:26:29.503246 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:29.503255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:29.503284 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:26:29.503414 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:29.503421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:29.503495 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:29.504546 (+ 1051us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:29.504649 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.505877 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.506943 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.507542 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.507659 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.510346 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:29.510350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:29.510350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:29.510362 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:29.512332 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.512773 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.512776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.512777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.512887 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.513099 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.513099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.513099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.513372 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:29.513523 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:29.513640 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 12:26:29.513850 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260928 12:26:29.536036 14818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:29.536569 14818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:29.541239 14818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Generated new TSK 0
I20260928 12:26:29.541445 14818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab.
Entry for principal oryx/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab.
I20260928 12:26:29.575719 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35971
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:29.656342 14824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:29.656569 14824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:29.656610 14824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:29.657790 14824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:29.657859 14824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:29.657892 14824 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:29.657932 14824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:29.659420 14824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:29.659689 14824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:29.659924 14824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:29.664868 14829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.664897 14830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.664921 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:29.664933 14824 server_base.cc:1034] running on GCE node
I20260928 12:26:29.665256 14824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:29.665504 14824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:29.666672 14824 hybrid_clock.cc:648] HybridClock initialized: now 1790598389666673 us; error 41 us; skew 500 ppm
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:29.669150 14824 init.cc:377] Logged in from keytab as oryx/127.7.19.65@KRBTEST.COM (short username oryx)
I20260928 12:26:29.669610 14824 webserver.cc:460] Webserver started at http://127.7.19.65:43485/ using document root <none> and password file <none>
I20260928 12:26:29.669832 14824 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:29.669906 14824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:29.670038 14824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:29.671099 14824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "2fc41ff9712a42e2aeb73feb801d6369"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "fe5432aa9f38db184a1bf28dde99414b"
server_key_iv: "d1e5f0f7052d5a265bfdc221951ac265"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.671458 14824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "2fc41ff9712a42e2aeb73feb801d6369"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "fe5432aa9f38db184a1bf28dde99414b"
server_key_iv: "d1e5f0f7052d5a265bfdc221951ac265"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.672802 14824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:29.673611 14839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:29.673770 14824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:29.673877 14824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "2fc41ff9712a42e2aeb73feb801d6369"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "fe5432aa9f38db184a1bf28dde99414b"
server_key_iv: "d1e5f0f7052d5a265bfdc221951ac265"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.673974 14824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:29.687441 14824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:29.689472 14824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:29.689626 14824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:29.690181 14824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:29.690230 14824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:29.690261 14824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:29.690284 14824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:29.697499 14824 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:38913
I20260928 12:26:29.697528 14954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:38913 every 8 connection(s)
I20260928 12:26:29.697841 14824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:29.699788 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.690561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:45497 (local address 127.7.19.126:35971)
0928 12:26:29.690633 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:29.690635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:29.690939 (+ 304us) server_negotiation.cc:407] Connection header received
0928 12:26:29.691399 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:29.691400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:29.691414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:29.691475 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:29.692200 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.692926 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.693962 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.694165 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.694247 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.697004 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:29.697006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:29.697006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:29.697013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:29.698612 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.699000 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.699002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.699002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.699059 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.699272 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.699272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.699273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.699377 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:29.699430 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:29.699657 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:26:29.699726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:29.699923 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.690546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.7.19.65:45497)
0928 12:26:29.690694 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:29.690728 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:29.691270 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:29.691548 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:29.691551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:29.691846 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:29.692073 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.692077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.693021 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.693022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.693839 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.693841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.694251 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.694252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.694384 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.694886 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:29.694896 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:29.696913 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:29.698682 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.698686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.698688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.698901 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.699126 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.699127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.699127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.699217 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.699433 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:29.699436 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:29.699549 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 12:26:29.699627 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:26:29.699738 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 12:26:29.700446 14955 heartbeater.cc:347] Connected to a master server at 127.7.19.126:35971
I20260928 12:26:29.700525 14955 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:29.700748 14955 heartbeater.cc:511] Master 127.7.19.126:35971 requested a full tablet report, sending...
I20260928 12:26:29.700779 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 14824
I20260928 12:26:29.700878 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:29.701133 7245 external_mini_cluster.cc:1428] Setting key d47e1880b512f1326031d8a7f4b36b61
I20260928 12:26:29.701328 14750 ts_manager.cc:194] Registered new tserver with Master: 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913)
I20260928 12:26:29.702278 14750 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:45497
I20260928 12:26:29.708817 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.701568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.7.19.65:43297)
0928 12:26:29.701751 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:29.701760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:29.702084 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:29.702268 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:29.702269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:29.702496 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:29.702558 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.702560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.703665 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.703666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.704398 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.704400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.704882 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.704884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.704954 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.705492 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:29.705497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:29.705779 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:29.707579 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.707580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.707580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.707861 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.708113 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.708114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.708115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.708176 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.708469 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:29.708470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:29.708509 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:26:29.708566 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:26:29.708647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":61,"threads_started":1}
I20260928 12:26:29.708853 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.701622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:43297 (local address 127.7.19.126:35971)
0928 12:26:29.701658 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:29.701660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:29.701838 (+ 178us) server_negotiation.cc:407] Connection header received
0928 12:26:29.702168 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:29.702169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:29.702180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:29.702209 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:29.702642 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.703461 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.704476 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.704791 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.704865 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.705865 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:29.705866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:29.705867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:29.705875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:29.707479 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.707958 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.707960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.707960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.708022 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.708249 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.708250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.708250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.708395 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:29.708469 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:29.708610 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 12:26:29.708720 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab.
Entry for principal oryx/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab.
I20260928 12:26:29.731436 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35971
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:29.811980 14964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:29.812189 14964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:29.812218 14964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:29.813410 14964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:29.813453 14964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:29.813473 14964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:29.813491 14964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:29.815021 14964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:29.815250 14964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:29.815491 14964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:29.820243 14970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.820267 14972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.820250 14969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:29.820547 14964 server_base.cc:1034] running on GCE node
I20260928 12:26:29.820770 14964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:29.820961 14964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:29.822106 14964 hybrid_clock.cc:648] HybridClock initialized: now 1790598389822095 us; error 27 us; skew 500 ppm
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:29.824553 14964 init.cc:377] Logged in from keytab as oryx/127.7.19.66@KRBTEST.COM (short username oryx)
I20260928 12:26:29.825053 14964 webserver.cc:460] Webserver started at http://127.7.19.66:43205/ using document root <none> and password file <none>
I20260928 12:26:29.825286 14964 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:29.825361 14964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:29.825506 14964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:29.826542 14964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "465dae58ab594882ba704e044efba81d"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "e647403a4c6ed848cfc316f50de6bbaf"
server_key_iv: "6b75f1e450424bd140d19e4274deb860"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.826896 14964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "465dae58ab594882ba704e044efba81d"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "e647403a4c6ed848cfc316f50de6bbaf"
server_key_iv: "6b75f1e450424bd140d19e4274deb860"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.828250 14964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:29.829069 14979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:29.829262 14964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:29.829398 14964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "465dae58ab594882ba704e044efba81d"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "e647403a4c6ed848cfc316f50de6bbaf"
server_key_iv: "6b75f1e450424bd140d19e4274deb860"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.829475 14964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:29.848337 14964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:29.850436 14964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:29.850606 14964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:29.851183 14964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:29.851244 14964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:29.851285 14964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:29.851317 14964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:29.858754 14964 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:44315
I20260928 12:26:29.858809 15094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:44315 every 8 connection(s)
I20260928 12:26:29.859166 14964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:29.861614 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.851616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:36397 (local address 127.7.19.126:35971)
0928 12:26:29.851682 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:29.851685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:29.852005 (+ 320us) server_negotiation.cc:407] Connection header received
0928 12:26:29.852798 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:29.852800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:29.852815 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:29.852915 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:29.853854 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.854438 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.855568 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.855783 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.855876 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.858390 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:29.858392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:29.858392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:29.858399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:29.860026 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.860453 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.860454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.860454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.860502 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.860985 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.860986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.860986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.861091 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:29.861146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:29.861446 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 12:26:29.861526 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:26:29.861788 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.851619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.7.19.66:36397)
0928 12:26:29.851826 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:29.851879 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:29.852644 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:29.853005 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:29.853010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:29.853420 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:29.853734 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.853742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.854548 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.854549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.855449 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.855451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.855896 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.855897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.856115 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.856764 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:29.856778 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:29.858294 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:29.860126 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.860131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.860133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.860383 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.860561 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.860562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.860562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.860659 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.861153 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:29.861158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:29.861294 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 12:26:29.861402 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:26:29.861562 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260928 12:26:29.862232 15095 heartbeater.cc:347] Connected to a master server at 127.7.19.126:35971
I20260928 12:26:29.862317 15095 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:29.862496 15095 heartbeater.cc:511] Master 127.7.19.126:35971 requested a full tablet report, sending...
I20260928 12:26:29.862957 14751 ts_manager.cc:194] Registered new tserver with Master: 465dae58ab594882ba704e044efba81d (127.7.19.66:44315)
I20260928 12:26:29.863436 14751 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:36397
I20260928 12:26:29.866405 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 14964
I20260928 12:26:29.866539 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:29.866751 7245 external_mini_cluster.cc:1428] Setting key cc6d6a106644f262e5e93cdf27cc9185
I20260928 12:26:29.868947 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.863239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.7.19.66:44151)
0928 12:26:29.863427 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:29.863436 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:29.863524 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:29.863688 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:29.863689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:29.863889 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:29.863965 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.863968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.864660 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.864661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.865198 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:29.865199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.865542 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.865543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:29.865589 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.865948 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:29.865952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:29.866209 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:29.867828 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.867829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.867829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.867998 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.868208 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:29.868209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:29.868209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:29.868265 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:29.868625 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:29.868627 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:29.868701 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 12:26:29.868784 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:26:29.868865 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":122,"threads_started":1}
I20260928 12:26:29.869020 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:29.863328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:44151 (local address 127.7.19.126:35971)
0928 12:26:29.863372 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:29.863374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:29.863510 (+ 136us) server_negotiation.cc:407] Connection header received
0928 12:26:29.863610 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:29.863611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:29.863622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:29.863686 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:29.864063 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.864571 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.865269 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:29.865475 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:29.865532 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:29.866331 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:29.866332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:29.866333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:29.866339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:29.867726 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.868083 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.868084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.868084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.868135 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:29.868335 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:29.868336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:29.868336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:29.868537 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:29.868617 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:29.868822 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 12:26:29.868900 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab.
Entry for principal oryx/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab.
I20260928 12:26:29.892143 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35971
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:29.975129 15104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:29.975355 15104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:29.975396 15104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:29.976559 15104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:29.976651 15104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:29.976681 15104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 12:26:29.976725 15104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:29.978379 15104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.19.126:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:29.978667 15104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:29.978904 15104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:29.984014 15104 server_base.cc:1034] running on GCE node
W20260928 12:26:29.983979 15109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.984036 15112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:29.983990 15110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:29.984413 15104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:29.984707 15104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:29.985862 15104 hybrid_clock.cc:648] HybridClock initialized: now 1790598389985827 us; error 51 us; skew 500 ppm
Sep 28 12:26:29 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:29.988189 15104 init.cc:377] Logged in from keytab as oryx/127.7.19.67@KRBTEST.COM (short username oryx)
I20260928 12:26:29.988592 15104 webserver.cc:460] Webserver started at http://127.7.19.67:39199/ using document root <none> and password file <none>
I20260928 12:26:29.988837 15104 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:29.988883 15104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:29.989042 15104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:29.990074 15104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "a9e7fc013c32483682dce7ffc46e40df"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "1fb4e52e3e34022cdcb4f46cc6541359"
server_key_iv: "889feeeb4c269a26c9680527a6ab925e"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.990422 15104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "a9e7fc013c32483682dce7ffc46e40df"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "1fb4e52e3e34022cdcb4f46cc6541359"
server_key_iv: "889feeeb4c269a26c9680527a6ab925e"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.991731 15104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:29.992404 15119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:29.992568 15104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:29.992661 15104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "a9e7fc013c32483682dce7ffc46e40df"
format_stamp: "Formatted at 2026-09-28 12:26:29 on dist-test-slave-cb7x"
server_key: "1fb4e52e3e34022cdcb4f46cc6541359"
server_key_iv: "889feeeb4c269a26c9680527a6ab925e"
server_key_version: "encryptionkey@0"
I20260928 12:26:29.992753 15104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:30.002293 15104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:30.004405 15104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:30.004560 15104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:30.005103 15104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:30.005168 15104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.005211 15104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:30.005239 15104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.011601 15104 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:44317
I20260928 12:26:30.011718 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:44317 every 8 connection(s)
I20260928 12:26:30.011992 15104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:30.015801 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.005586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:50173 (local address 127.7.19.126:35971)
0928 12:26:30.005659 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.005662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.005842 (+ 180us) server_negotiation.cc:407] Connection header received
0928 12:26:30.006432 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.006434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.006447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.006496 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.007377 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.008212 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.009823 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.010028 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.010106 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.012765 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:30.012766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:30.012766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:30.012774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:30.014386 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.014801 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.014803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.014803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.014865 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.015134 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.015134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.015135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.015246 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:30.015360 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.015620 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.015703 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:30.015969 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.005575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.7.19.67:50173)
0928 12:26:30.005735 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.005772 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.006332 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.006562 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.006566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.006971 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:30.007252 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.007258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.008322 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.008323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.009734 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.009737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.010110 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.010111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.010257 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.010808 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:30.010818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:30.012622 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:30.014480 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:30.014486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:30.014488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:30.014702 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:30.014940 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:30.014941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:30.014941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:30.015056 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:30.015351 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:30.015356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:30.015493 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.015599 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.015746 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 12:26:30.016414 15235 heartbeater.cc:347] Connected to a master server at 127.7.19.126:35971
I20260928 12:26:30.016525 15235 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:30.016511 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15104
I20260928 12:26:30.016602 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:30.016721 15235 heartbeater.cc:511] Master 127.7.19.126:35971 requested a full tablet report, sending...
I20260928 12:26:30.016860 7245 external_mini_cluster.cc:1428] Setting key 359ecf04141e2806f69ede46ec7e3973
I20260928 12:26:30.017071 14750 ts_manager.cc:194] Registered new tserver with Master: a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.017555 14750 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:50173
I20260928 12:26:30.017652 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:26:30.023449 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.017483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:56877 (local address 127.7.19.126:35971)
0928 12:26:30.017516 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.017517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.017632 (+ 115us) server_negotiation.cc:407] Connection header received
0928 12:26:30.017705 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.017706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.017717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.017772 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.018186 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.018891 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.019812 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.019988 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.020049 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.021039 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:30.021040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:30.021040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:30.021045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:30.022445 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.022817 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.022817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.022817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.022862 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.023060 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.023061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.023061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.023146 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:30.023193 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.023345 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.023395 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:26:30.023597 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.017442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.7.19.67:56877)
0928 12:26:30.017577 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.017587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.017655 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.017777 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.017778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.018007 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:30.018068 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.018070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.018985 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.018985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.019672 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.019675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.020066 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.020067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.020136 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.020540 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:30.020545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:30.020950 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:30.022508 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:30.022510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:30.022510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:30.022730 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:30.022926 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:30.022927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:30.022927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:30.023000 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:30.023206 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:30.023207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:30.023254 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.023330 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.023451 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:26:30.039003 14804 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:30.034038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.7.19.126:35971)
0928 12:26:30.034073 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.034075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.034114 (+ 39us) server_negotiation.cc:407] Connection header received
0928 12:26:30.034166 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.034167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.034174 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.034221 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.034754 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.035201 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.035891 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.036058 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.036108 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.038905 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39228: BlockingRecv error: recv got EOF from 127.0.0.1:39228 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:30.046047 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.039715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.7.19.126:35971)
0928 12:26:30.039791 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.039793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.039803 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:30.039851 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.039851 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.039859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.039906 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.040331 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.040847 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.041646 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.041892 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.041948 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.043764 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:30.043765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:30.043766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:30.043771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:30.045144 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.045417 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.045418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.045418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.045463 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.045600 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.045600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.045600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.045679 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:30.045743 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.045851 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.045902 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 12:26:30.047626 14750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:30.049172 14750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:30.055193 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.051226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:44315 (local address 127.0.0.1:51716)
0928 12:26:30.051592 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.051599 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.051631 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.051879 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.051883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.051886 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.051971 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.051973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.053276 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.053277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.054117 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.054119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.054935 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.054936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.055005 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.055010 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.055039 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.055109 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260928 12:26:30.055202 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.051294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.7.19.65:38913)
0928 12:26:30.051727 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.051731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.051738 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:26:30.051788 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.051790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.051801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.051883 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.052056 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.052839 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.054121 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.054760 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.054844 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.054916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.055058 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.055103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":90,"threads_started":1}
I20260928 12:26:30.055251 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.051192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38913 (local address 127.0.0.1:54062)
0928 12:26:30.051357 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.051397 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.051489 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.051897 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.051898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.051900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.051972 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.051974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.052927 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.052928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.054031 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.054033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.054842 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.054843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.054919 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.054942 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.055042 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.055133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":48,"thread_start_us":45,"threads_started":1}
I20260928 12:26:30.055626 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.051226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:44317 (local address 127.0.0.1:32860)
0928 12:26:30.051463 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.051472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.051529 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.052157 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.052158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.052160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.052192 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.052193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.053389 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.053390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.054431 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.054434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.055415 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.055416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.055479 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.055482 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.055531 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.055562 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260928 12:26:30.055634 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.051462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.7.19.66:44315)
0928 12:26:30.051624 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.051630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.051643 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:26:30.051702 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.051703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.051716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.051828 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.052061 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.053175 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.054203 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.054837 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.055393 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.055466 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.055490 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.055540 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260928 12:26:30.055920 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.051433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.7.19.67:44317)
0928 12:26:30.051945 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.051950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.051958 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:30.051991 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.051993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.052009 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.052080 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.052275 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.053302 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.054514 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.055240 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.055309 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.055381 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.055775 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.055812 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":101,"threads_started":1}
I20260928 12:26:30.056227 14889 tablet_service.cc:1467] Processing CreateTablet for tablet f76f5a35a7f34e878b715a31208306eb (DEFAULT_TABLE table=test-table [id=876790bbfada453bbaa0e4eb39f646de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.056460 15169 tablet_service.cc:1467] Processing CreateTablet for tablet f76f5a35a7f34e878b715a31208306eb (DEFAULT_TABLE table=test-table [id=876790bbfada453bbaa0e4eb39f646de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.056560 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76f5a35a7f34e878b715a31208306eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.056777 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76f5a35a7f34e878b715a31208306eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.059127 15263 tablet_bootstrap.cc:492] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Bootstrap starting.
I20260928 12:26:30.059685 15264 tablet_bootstrap.cc:492] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Bootstrap starting.
I20260928 12:26:30.060153 15263 tablet_bootstrap.cc:654] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.060213 15264 tablet_bootstrap.cc:654] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.060124 15029 tablet_service.cc:1467] Processing CreateTablet for tablet f76f5a35a7f34e878b715a31208306eb (DEFAULT_TABLE table=test-table [id=876790bbfada453bbaa0e4eb39f646de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.060429 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f76f5a35a7f34e878b715a31208306eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.060464 15263 log.cc:824] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:30.060525 15264 log.cc:824] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:30.061249 15264 tablet_bootstrap.cc:492] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: No bootstrap required, opened a new log
I20260928 12:26:30.061379 15264 ts_tablet_manager.cc:1397] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:30.062889 15264 raft_consensus.cc:348] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.063088 15264 raft_consensus.cc:374] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.063131 15264 raft_consensus.cc:729] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.063262 15264 consensus_queue.cc:260] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.063552 15264 ts_tablet_manager.cc:1428] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:30.063589 14955 heartbeater.cc:503] Master 127.7.19.126:35971 was elected leader, sending a full tablet report...
I20260928 12:26:30.063771 15268 tablet_bootstrap.cc:492] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Bootstrap starting.
I20260928 12:26:30.064373 15268 tablet_bootstrap.cc:654] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.064414 15263 tablet_bootstrap.cc:492] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: No bootstrap required, opened a new log
I20260928 12:26:30.064466 15263 ts_tablet_manager.cc:1397] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 12:26:30.064751 15268 log.cc:824] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:30.065433 15268 tablet_bootstrap.cc:492] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: No bootstrap required, opened a new log
I20260928 12:26:30.065501 15268 ts_tablet_manager.cc:1397] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:30.066514 15263 raft_consensus.cc:348] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.066673 15263 raft_consensus.cc:374] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.066709 15263 raft_consensus.cc:729] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e7fc013c32483682dce7ffc46e40df, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.066825 15263 consensus_queue.cc:260] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.067068 15263 ts_tablet_manager.cc:1428] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:26:30.067111 15235 heartbeater.cc:503] Master 127.7.19.126:35971 was elected leader, sending a full tablet report...
I20260928 12:26:30.067490 15268 raft_consensus.cc:348] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.067642 15268 raft_consensus.cc:374] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.067683 15268 raft_consensus.cc:729] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465dae58ab594882ba704e044efba81d, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.067791 15268 consensus_queue.cc:260] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.068082 15268 ts_tablet_manager.cc:1428] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:26:30.068142 15095 heartbeater.cc:503] Master 127.7.19.126:35971 was elected leader, sending a full tablet report...
W20260928 12:26:30.109882 15096 tablet.cc:2369] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:26:30.198669 14956 tablet.cc:2369] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:26:30.263023 15236 tablet.cc:2369] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:30.307723 15271 raft_consensus.cc:482] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:30.307847 15271 raft_consensus.cc:504] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.308192 15271 leader_election.cc:288] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.311532 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.308452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:60425 (local address 127.7.19.65:38913)
0928 12:26:30.308497 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.308499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.308582 (+ 83us) server_negotiation.cc:407] Connection header received
0928 12:26:30.308650 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.308652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.308665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.308759 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.308936 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.309467 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.310660 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.311081 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.311135 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.311143 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.311411 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.311460 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:30.311599 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.308474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:42549 (local address 127.7.19.67:44317)
0928 12:26:30.308538 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.308539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.308786 (+ 247us) server_negotiation.cc:407] Connection header received
0928 12:26:30.308821 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.308822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.308834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.308898 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.309101 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.309791 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.310795 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.311225 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.311302 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.311314 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.311482 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.311527 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:30.311586 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.308418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:44317 (local address 127.7.19.66:42549)
0928 12:26:30.308710 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.308720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.308777 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.308930 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.308930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.308932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.308999 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.309001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.309882 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.309883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.310702 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.310704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.311312 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.311312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.311363 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.311391 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.311450 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.311484 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":89,"thread_start_us":104,"threads_started":1}
I20260928 12:26:30.311595 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.308413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38913 (local address 127.7.19.66:60425)
0928 12:26:30.308503 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.308515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.308578 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.308784 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.308785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.308787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.308863 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.308864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.309545 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.309546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.310573 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.310575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.311165 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.311166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.311326 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.311331 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.311396 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.311454 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 12:26:30.312029 14908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76f5a35a7f34e878b715a31208306eb" candidate_uuid: "465dae58ab594882ba704e044efba81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc41ff9712a42e2aeb73feb801d6369" is_pre_election: true
I20260928 12:26:30.312034 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76f5a35a7f34e878b715a31208306eb" candidate_uuid: "465dae58ab594882ba704e044efba81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df" is_pre_election: true
I20260928 12:26:30.312216 15189 raft_consensus.cc:2454] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465dae58ab594882ba704e044efba81d in term 0.
I20260928 12:26:30.312209 14908 raft_consensus.cc:2454] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465dae58ab594882ba704e044efba81d in term 0.
I20260928 12:26:30.312841 14980 leader_election.cc:302] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, 465dae58ab594882ba704e044efba81d; no voters:
I20260928 12:26:30.313036 15271 raft_consensus.cc:2781] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:30.313117 15271 raft_consensus.cc:482] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:30.313158 15271 raft_consensus.cc:3037] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.313959 15271 raft_consensus.cc:504] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.314121 15271 leader_election.cc:288] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [CANDIDATE]: Term 1 election: Requested vote from peers 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.314260 14908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76f5a35a7f34e878b715a31208306eb" candidate_uuid: "465dae58ab594882ba704e044efba81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc41ff9712a42e2aeb73feb801d6369"
I20260928 12:26:30.314327 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f76f5a35a7f34e878b715a31208306eb" candidate_uuid: "465dae58ab594882ba704e044efba81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df"
I20260928 12:26:30.314350 14908 raft_consensus.cc:3037] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.314395 15189 raft_consensus.cc:3037] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.315243 14908 raft_consensus.cc:2454] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465dae58ab594882ba704e044efba81d in term 1.
I20260928 12:26:30.315261 15189 raft_consensus.cc:2454] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465dae58ab594882ba704e044efba81d in term 1.
I20260928 12:26:30.315450 14980 leader_election.cc:302] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, 465dae58ab594882ba704e044efba81d; no voters:
I20260928 12:26:30.315542 15271 raft_consensus.cc:2781] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:30.315690 15271 raft_consensus.cc:686] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 1 LEADER]: Becoming Leader. State: Replica: 465dae58ab594882ba704e044efba81d, State: Running, Role: LEADER
I20260928 12:26:30.315824 15271 consensus_queue.cc:237] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } }
I20260928 12:26:30.316764 14751 catalog_manager.cc:5495] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d reported cstate change: term changed from 0 to 1, leader changed from <none> to 465dae58ab594882ba704e044efba81d (127.7.19.66). New cstate: current_term: 1 leader_uuid: "465dae58ab594882ba704e044efba81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } health_report { overall_health: HEALTHY } } }
I20260928 12:26:30.337486 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.334933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.7.19.66:44315)
0928 12:26:30.335014 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.335016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.335026 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:30.335088 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.335089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.335111 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.335166 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:30.335276 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.335739 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.336472 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.336750 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.336835 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.337047 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:30.337156 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:30.337299 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.337316 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.337368 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 12:26:30.339278 14908 raft_consensus.cc:1261] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Refusing update from remote peer 465dae58ab594882ba704e044efba81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:30.339354 15189 raft_consensus.cc:1261] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Refusing update from remote peer 465dae58ab594882ba704e044efba81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:30.339673 15273 consensus_queue.cc:1035] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:30.339807 15271 consensus_queue.cc:1035] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:30.342062 15276 mvcc.cc:205] Tried to move back new op lower bound from 7334291006826962944 to 7334291006734258176. Current Snapshot: MvccSnapshot[applied={T|T < 7334291006826962944}]
I20260928 12:26:30.342272 15278 mvcc.cc:205] Tried to move back new op lower bound from 7334291006826962944 to 7334291006734258176. Current Snapshot: MvccSnapshot[applied={T|T < 7334291006826962944}]
I20260928 12:26:30.342437 15277 mvcc.cc:205] Tried to move back new op lower bound from 7334291006826962944 to 7334291006734258176. Current Snapshot: MvccSnapshot[applied={T|T < 7334291006826962944}]
I20260928 12:26:30.344583 14751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:30.344691 14751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:30.345446 14751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Sending DeleteTablet for 3 replicas of tablet f76f5a35a7f34e878b715a31208306eb
I20260928 12:26:30.345768 15029 tablet_service.cc:1514] Processing DeleteTablet for tablet f76f5a35a7f34e878b715a31208306eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:30 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260928 12:26:30.345781 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet f76f5a35a7f34e878b715a31208306eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:30 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:32860
I20260928 12:26:30.345965 14889 tablet_service.cc:1514] Processing DeleteTablet for tablet f76f5a35a7f34e878b715a31208306eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:30 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:54062
I20260928 12:26:30.346007 14751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:30.346153 14751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:30.346211 15287 tablet_replica.cc:330] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: stopping tablet replica
I20260928 12:26:30.346235 15286 tablet_replica.cc:330] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: stopping tablet replica
I20260928 12:26:30.346292 15287 raft_consensus.cc:2229] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:30.346421 15287 raft_consensus.cc:2258] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:30.346455 15286 raft_consensus.cc:2229] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:30.346633 15286 raft_consensus.cc:2258] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:30.347286 15287 ts_tablet_manager.cc:1905] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:30.347429 14889 tablet_service.cc:1467] Processing CreateTablet for tablet c91e05ad660c4f419e02ed2c0e0dfde9 (DEFAULT_TABLE table=test-table [id=6a0bb8e654c6416488fa734dac552c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.347517 15169 tablet_service.cc:1467] Processing CreateTablet for tablet c91e05ad660c4f419e02ed2c0e0dfde9 (DEFAULT_TABLE table=test-table [id=6a0bb8e654c6416488fa734dac552c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.347543 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91e05ad660c4f419e02ed2c0e0dfde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.347605 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91e05ad660c4f419e02ed2c0e0dfde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.348780 15263 tablet_bootstrap.cc:492] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Bootstrap starting.
I20260928 12:26:30.348851 15264 tablet_bootstrap.cc:492] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Bootstrap starting.
I20260928 12:26:30.349207 15264 tablet_bootstrap.cc:654] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.349305 15263 tablet_bootstrap.cc:654] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.349804 15287 ts_tablet_manager.cc:1918] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:30.349875 15287 log.cc:1196] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/f76f5a35a7f34e878b715a31208306eb
I20260928 12:26:30.349949 15264 tablet_bootstrap.cc:492] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: No bootstrap required, opened a new log
I20260928 12:26:30.349989 15264 ts_tablet_manager.cc:1397] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:30.350072 15287 ts_tablet_manager.cc:1939] T f76f5a35a7f34e878b715a31208306eb P a9e7fc013c32483682dce7ffc46e40df: Deleting consensus metadata
I20260928 12:26:30.350116 15264 raft_consensus.cc:348] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.350175 15264 raft_consensus.cc:374] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.350194 15264 raft_consensus.cc:729] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.350234 15264 consensus_queue.cc:260] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.350248 15288 tablet_replica.cc:330] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: stopping tablet replica
I20260928 12:26:30.350335 15264 ts_tablet_manager.cc:1428] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.350343 15288 raft_consensus.cc:2229] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:30.350464 15288 raft_consensus.cc:2258] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:30.350461 15029 tablet_service.cc:1467] Processing CreateTablet for tablet c91e05ad660c4f419e02ed2c0e0dfde9 (DEFAULT_TABLE table=test-table [id=6a0bb8e654c6416488fa734dac552c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.350518 14736 catalog_manager.cc:4841] TS a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317): tablet f76f5a35a7f34e878b715a31208306eb (table test-table [id=876790bbfada453bbaa0e4eb39f646de]) successfully deleted
I20260928 12:26:30.350556 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91e05ad660c4f419e02ed2c0e0dfde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.350836 15288 ts_tablet_manager.cc:1905] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:30.351853 15268 tablet_bootstrap.cc:492] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Bootstrap starting.
I20260928 12:26:30.352232 15268 tablet_bootstrap.cc:654] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.352375 15286 ts_tablet_manager.cc:1905] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:30.352582 15288 ts_tablet_manager.cc:1918] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:30.352648 15288 log.cc:1196] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/f76f5a35a7f34e878b715a31208306eb
I20260928 12:26:30.352972 15288 ts_tablet_manager.cc:1939] T f76f5a35a7f34e878b715a31208306eb P 2fc41ff9712a42e2aeb73feb801d6369: Deleting consensus metadata
I20260928 12:26:30.353375 14735 catalog_manager.cc:4841] TS 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913): tablet f76f5a35a7f34e878b715a31208306eb (table test-table [id=876790bbfada453bbaa0e4eb39f646de]) successfully deleted
I20260928 12:26:30.353756 15286 ts_tablet_manager.cc:1918] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:30.353824 15286 log.cc:1196] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/f76f5a35a7f34e878b715a31208306eb
I20260928 12:26:30.354072 15286 ts_tablet_manager.cc:1939] T f76f5a35a7f34e878b715a31208306eb P 465dae58ab594882ba704e044efba81d: Deleting consensus metadata
I20260928 12:26:30.354418 14738 catalog_manager.cc:4841] TS 465dae58ab594882ba704e044efba81d (127.7.19.66:44315): tablet f76f5a35a7f34e878b715a31208306eb (table test-table [id=876790bbfada453bbaa0e4eb39f646de]) successfully deleted
I20260928 12:26:30.354645 15263 tablet_bootstrap.cc:492] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: No bootstrap required, opened a new log
I20260928 12:26:30.354708 15263 ts_tablet_manager.cc:1397] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 12:26:30.354877 15263 raft_consensus.cc:348] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.354947 15263 raft_consensus.cc:374] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.354961 15263 raft_consensus.cc:729] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e7fc013c32483682dce7ffc46e40df, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.355011 15263 consensus_queue.cc:260] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.355100 15263 ts_tablet_manager.cc:1428] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.360101 15268 tablet_bootstrap.cc:492] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: No bootstrap required, opened a new log
I20260928 12:26:30.360143 15268 ts_tablet_manager.cc:1397] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260928 12:26:30.360335 15268 raft_consensus.cc:348] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.360406 15268 raft_consensus.cc:374] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.360430 15268 raft_consensus.cc:729] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465dae58ab594882ba704e044efba81d, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.360492 15268 consensus_queue.cc:260] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.360574 15268 ts_tablet_manager.cc:1428] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:30.552163 15267 raft_consensus.cc:482] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:30.552266 15267 raft_consensus.cc:504] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.552600 15267 leader_election.cc:288] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 465dae58ab594882ba704e044efba81d (127.7.19.66:44315), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.555711 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.552827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:44315 (local address 127.7.19.65:58487)
0928 12:26:30.552958 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.552969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.553057 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.553250 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.553250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.553252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.553324 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.553326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.553948 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.553949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.554799 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.554801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.555442 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.555443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.555527 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.555532 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.555587 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.555621 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":23,"thread_start_us":61,"threads_started":1}
I20260928 12:26:30.555721 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.552894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:58487 (local address 127.7.19.66:44315)
0928 12:26:30.552948 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.552950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.553043 (+ 93us) server_negotiation.cc:407] Connection header received
0928 12:26:30.553150 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.553155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.553163 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.553237 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.553390 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.553866 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.554924 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.555334 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.555397 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.555406 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.555608 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.555660 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:26:30.555754 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.552896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:57543 (local address 127.7.19.67:44317)
0928 12:26:30.552962 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.552964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.553040 (+ 76us) server_negotiation.cc:407] Connection header received
0928 12:26:30.553137 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.553138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.553147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.553196 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:30.553417 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.553866 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.554952 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.555310 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.555397 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.555407 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.555651 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.555690 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:26:30.555930 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.552832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:44317 (local address 127.7.19.65:57543)
0928 12:26:30.552958 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.552969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.553057 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.553226 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.553227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.553229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:30.553312 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.553319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.553968 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.553969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.554814 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.554817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.555469 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.555471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.555546 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.555549 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.555817 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.555862 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":50,"thread_start_us":65,"threads_started":1}
I20260928 12:26:30.556232 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91e05ad660c4f419e02ed2c0e0dfde9" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df" is_pre_election: true
I20260928 12:26:30.556219 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91e05ad660c4f419e02ed2c0e0dfde9" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d" is_pre_election: true
I20260928 12:26:30.556324 15189 raft_consensus.cc:2454] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 0.
I20260928 12:26:30.556339 15049 raft_consensus.cc:2454] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 0.
I20260928 12:26:30.556520 14841 leader_election.cc:302] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:30.556664 15267 raft_consensus.cc:2781] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:30.556768 15267 raft_consensus.cc:482] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:30.556802 15267 raft_consensus.cc:3037] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.557483 15267 raft_consensus.cc:504] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.557639 15267 leader_election.cc:288] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 election: Requested vote from peers 465dae58ab594882ba704e044efba81d (127.7.19.66:44315), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.557778 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91e05ad660c4f419e02ed2c0e0dfde9" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d"
I20260928 12:26:30.557781 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91e05ad660c4f419e02ed2c0e0dfde9" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df"
I20260928 12:26:30.557853 15049 raft_consensus.cc:3037] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.557874 15189 raft_consensus.cc:3037] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.558447 15189 raft_consensus.cc:2454] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 1.
I20260928 12:26:30.558497 15049 raft_consensus.cc:2454] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 1.
I20260928 12:26:30.558650 14841 leader_election.cc:302] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:30.558768 15267 raft_consensus.cc:2781] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:30.558898 15267 raft_consensus.cc:686] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Running, Role: LEADER
I20260928 12:26:30.558981 15267 consensus_queue.cc:237] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.559644 14751 catalog_manager.cc:5495] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } health_report { overall_health: UNKNOWN } } }
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598389, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.126@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:30.608352 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.601439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.7.19.126:35971)
0928 12:26:30.601563 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.601565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.601667 (+ 102us) server_negotiation.cc:407] Connection header received
0928 12:26:30.601714 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.601715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.601726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.601790 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.602225 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.602827 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.603608 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.603797 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.603850 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.605867 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:30.605869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:30.605869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:30.605875 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:30.607326 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.607681 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.607682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.607682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.607733 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.607898 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.607899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.607899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.607992 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:30.608040 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.608154 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.608241 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260928 12:26:30.608409 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.601418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35971 (local address 127.0.0.1:39252)
0928 12:26:30.601589 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.601600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.601668 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.601806 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.601807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.602106 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:30.602158 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.602159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.602899 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.602900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.603508 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.603510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.603861 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.603862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.603902 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.604381 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:30.604385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:30.605791 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:30.607417 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:30.607423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:30.607425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:30.607616 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:30.607790 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:30.607791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:30.607791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:30.607845 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:30.608045 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:30.608050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:30.608088 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.608140 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.608288 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":70,"thread_start_us":87,"threads_started":1}
I20260928 12:26:30.610455 14751 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:39252:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 12:26:30.610649 14751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:30.612044 15029 tablet_service.cc:1467] Processing CreateTablet for tablet dfe42a67dc7c4b74a86768d67646c324 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=537fc3542d5c4d7fb62067cee5337598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:30.612079 14889 tablet_service.cc:1467] Processing CreateTablet for tablet dfe42a67dc7c4b74a86768d67646c324 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=537fc3542d5c4d7fb62067cee5337598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:30.612206 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe42a67dc7c4b74a86768d67646c324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.612211 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe42a67dc7c4b74a86768d67646c324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.612177 15169 tablet_service.cc:1467] Processing CreateTablet for tablet dfe42a67dc7c4b74a86768d67646c324 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=537fc3542d5c4d7fb62067cee5337598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 12:26:30.612318 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe42a67dc7c4b74a86768d67646c324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.613595 15263 tablet_bootstrap.cc:492] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: Bootstrap starting.
I20260928 12:26:30.613700 15264 tablet_bootstrap.cc:492] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: Bootstrap starting.
I20260928 12:26:30.613950 15263 tablet_bootstrap.cc:654] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.614055 15264 tablet_bootstrap.cc:654] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.614411 15268 tablet_bootstrap.cc:492] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: Bootstrap starting.
I20260928 12:26:30.614578 15263 tablet_bootstrap.cc:492] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: No bootstrap required, opened a new log
I20260928 12:26:30.614617 15263 ts_tablet_manager.cc:1397] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:30.614691 15268 tablet_bootstrap.cc:654] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.614739 15264 tablet_bootstrap.cc:492] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: No bootstrap required, opened a new log
I20260928 12:26:30.614737 15263 raft_consensus.cc:348] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.614784 15263 raft_consensus.cc:374] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.614799 15263 raft_consensus.cc:729] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e7fc013c32483682dce7ffc46e40df, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.614784 15264 ts_tablet_manager.cc:1397] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:30.614856 15263 consensus_queue.cc:260] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.614979 15263 ts_tablet_manager.cc:1428] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.614944 15264 raft_consensus.cc:348] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.615024 15264 raft_consensus.cc:374] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.615047 15264 raft_consensus.cc:729] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.615008 15270 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.615090 15270 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.615095 15264 consensus_queue.cc:260] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.615188 15264 ts_tablet_manager.cc:1428] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:30.615298 15267 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.615362 15267 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.615890 15268 tablet_bootstrap.cc:492] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: No bootstrap required, opened a new log
I20260928 12:26:30.615952 15268 ts_tablet_manager.cc:1397] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:26:30.616063 15268 raft_consensus.cc:348] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.616133 15268 raft_consensus.cc:374] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.616148 15268 raft_consensus.cc:729] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465dae58ab594882ba704e044efba81d, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.616197 15268 consensus_queue.cc:260] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.616292 15268 ts_tablet_manager.cc:1428] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.616308 15271 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.616402 15271 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.722779 15267 raft_consensus.cc:482] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:30.722898 15267 raft_consensus.cc:504] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.723069 15267 leader_election.cc:288] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 465dae58ab594882ba704e044efba81d (127.7.19.66:44315), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.723289 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe42a67dc7c4b74a86768d67646c324" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df" is_pre_election: true
I20260928 12:26:30.723413 15189 raft_consensus.cc:2454] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 0.
I20260928 12:26:30.723560 14841 leader_election.cc:302] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:30.723798 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe42a67dc7c4b74a86768d67646c324" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d" is_pre_election: true
I20260928 12:26:30.723942 15049 raft_consensus.cc:2454] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 0.
I20260928 12:26:30.724107 15267 raft_consensus.cc:2781] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:30.724149 15267 raft_consensus.cc:482] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:30.724170 15267 raft_consensus.cc:3037] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.725054 15267 raft_consensus.cc:504] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.725206 15267 leader_election.cc:288] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 election: Requested vote from peers 465dae58ab594882ba704e044efba81d (127.7.19.66:44315), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:30.725363 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe42a67dc7c4b74a86768d67646c324" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d"
I20260928 12:26:30.725426 15049 raft_consensus.cc:3037] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.725553 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe42a67dc7c4b74a86768d67646c324" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df"
I20260928 12:26:30.725657 15189 raft_consensus.cc:3037] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:30.725986 15049 raft_consensus.cc:2454] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 1.
I20260928 12:26:30.726142 14843 leader_election.cc:302] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, 465dae58ab594882ba704e044efba81d; no voters:
I20260928 12:26:30.726233 15267 raft_consensus.cc:2781] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:30.726280 15267 raft_consensus.cc:686] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Running, Role: LEADER
I20260928 12:26:30.726337 15267 consensus_queue.cc:237] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.726401 15189 raft_consensus.cc:2454] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 1.
I20260928 12:26:30.726662 15267 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: TxnStatusTablet state changed. Reason: New leader 2fc41ff9712a42e2aeb73feb801d6369. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.726735 15267 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:30.726940 15307 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 12:26:30.727057 15307 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 12:26:30.727142 14750 catalog_manager.cc:5495] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:30.733129 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.730326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.7.19.65:38913)
0928 12:26:30.730374 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.730376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.730387 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:26:30.730457 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.730458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.730477 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.730560 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:30.730663 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.731406 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.732295 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.732490 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.732567 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.732799 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:30.732898 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:30.732984 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.733042 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.733065 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:26:30.733167 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.730269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38913 (local address 127.0.0.1:54076)
0928 12:26:30.730352 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.730362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.730398 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.730559 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.730559 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.730560 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:30.730605 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.730607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.731502 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.731503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.732168 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.732170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.732565 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.732567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.732644 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.732708 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:30.732971 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:30.732980 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.733036 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.733089 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 12:26:30.734252 15189 raft_consensus.cc:1261] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Refusing update from remote peer 2fc41ff9712a42e2aeb73feb801d6369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:30.734321 15049 raft_consensus.cc:1261] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Refusing update from remote peer 2fc41ff9712a42e2aeb73feb801d6369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:30.734503 15267 consensus_queue.cc:1035] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:30.734789 15267 consensus_queue.cc:1035] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:30.735248 15270 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: TxnStatusTablet state changed. Reason: New leader 2fc41ff9712a42e2aeb73feb801d6369. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.735351 15270 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.735332 15271 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: TxnStatusTablet state changed. Reason: New leader 2fc41ff9712a42e2aeb73feb801d6369. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.735423 15271 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.735935 15267 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.736032 15267 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:30.736126 15267 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.736172 15267 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 2fc41ff9712a42e2aeb73feb801d6369: This TxnStatusTablet replica's current role is: LEADER
I20260928 12:26:30.736433 15271 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.736509 15271 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P 465dae58ab594882ba704e044efba81d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.736510 15270 tablet_replica.cc:439] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } } }
I20260928 12:26:30.736577 15270 tablet_replica.cc:442] T dfe42a67dc7c4b74a86768d67646c324 P a9e7fc013c32483682dce7ffc46e40df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 12:26:30.740018 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.737755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.7.19.65:38913)
0928 12:26:30.737800 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.737802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.737809 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:26:30.737874 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.737874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.737882 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.737949 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:30.738069 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.738523 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.739362 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.739563 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.739641 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.739783 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:30.739815 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:30.739857 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.739930 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.739948 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:26:30.744736 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.742331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:52509 (local address 127.7.19.65:38913)
0928 12:26:30.742372 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.742374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.742522 (+ 148us) server_negotiation.cc:407] Connection header received
0928 12:26:30.742592 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.742592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.742601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.742653 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:30.742798 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.743223 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.743998 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.744178 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.744246 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.744434 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:30.744469 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:30.744523 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.744591 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.744624 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":22}
I20260928 12:26:30.744753 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.742275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38913 (local address 127.7.19.65:52509)
0928 12:26:30.742447 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:30.742456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:30.742536 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:30.742656 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:30.742656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:30.742657 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 12:26:30.742721 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.742722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.743307 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.743307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.743905 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:30.743906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.744275 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.744276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:30.744333 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.744344 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 12:26:30.744538 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 12:26:30.744539 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 12:26:30.744587 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:26:30.744642 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260928 12:26:30.746500 15189 raft_consensus.cc:1261] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Refusing update from remote peer 2fc41ff9712a42e2aeb73feb801d6369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:30.746642 15049 raft_consensus.cc:1261] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Refusing update from remote peer 2fc41ff9712a42e2aeb73feb801d6369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:30.746749 15294 consensus_queue.cc:1035] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:30.746850 15267 consensus_queue.cc:1035] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:30.758074 14750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:30.758169 14750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:30.758733 14750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Sending DeleteTablet for 3 replicas of tablet c91e05ad660c4f419e02ed2c0e0dfde9
I20260928 12:26:30.759027 14889 tablet_service.cc:1514] Processing DeleteTablet for tablet c91e05ad660c4f419e02ed2c0e0dfde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:30 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:54062
I20260928 12:26:30.759039 15029 tablet_service.cc:1514] Processing DeleteTablet for tablet c91e05ad660c4f419e02ed2c0e0dfde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:30 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260928 12:26:30.759124 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet c91e05ad660c4f419e02ed2c0e0dfde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:30 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:32860
I20260928 12:26:30.759145 15288 tablet_replica.cc:330] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: stopping tablet replica
I20260928 12:26:30.759187 15288 raft_consensus.cc:2229] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:30.759204 15286 tablet_replica.cc:330] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: stopping tablet replica
I20260928 12:26:30.759254 15286 raft_consensus.cc:2229] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:30.759272 15288 raft_consensus.cc:2258] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:30.759315 15286 raft_consensus.cc:2258] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:30.759378 15287 tablet_replica.cc:330] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: stopping tablet replica
I20260928 12:26:30.759429 15287 raft_consensus.cc:2229] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:30.759491 15287 raft_consensus.cc:2258] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:30.760775 15288 ts_tablet_manager.cc:1905] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:30.762463 15288 ts_tablet_manager.cc:1918] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:30.762539 15288 log.cc:1196] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/c91e05ad660c4f419e02ed2c0e0dfde9
I20260928 12:26:30.762756 15288 ts_tablet_manager.cc:1939] T c91e05ad660c4f419e02ed2c0e0dfde9 P 2fc41ff9712a42e2aeb73feb801d6369: Deleting consensus metadata
I20260928 12:26:30.763058 14735 catalog_manager.cc:4841] TS 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913): tablet c91e05ad660c4f419e02ed2c0e0dfde9 (table test-table [id=6a0bb8e654c6416488fa734dac552c67]) successfully deleted
I20260928 12:26:30.764187 15287 ts_tablet_manager.cc:1905] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:30.765431 15287 ts_tablet_manager.cc:1918] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:30.765506 15287 log.cc:1196] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/c91e05ad660c4f419e02ed2c0e0dfde9
I20260928 12:26:30.765704 15287 ts_tablet_manager.cc:1939] T c91e05ad660c4f419e02ed2c0e0dfde9 P a9e7fc013c32483682dce7ffc46e40df: Deleting consensus metadata
I20260928 12:26:30.766026 14736 catalog_manager.cc:4841] TS a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317): tablet c91e05ad660c4f419e02ed2c0e0dfde9 (table test-table [id=6a0bb8e654c6416488fa734dac552c67]) successfully deleted
W20260928 12:26:30.768867 15316 consensus_queue.cc:1159] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "465dae58ab594882ba704e044efba81d" status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20260928 12:26:30.769146 15286 ts_tablet_manager.cc:1905] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:30.770617 15286 ts_tablet_manager.cc:1918] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:30.770699 15286 log.cc:1196] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/c91e05ad660c4f419e02ed2c0e0dfde9
I20260928 12:26:30.770920 15286 ts_tablet_manager.cc:1939] T c91e05ad660c4f419e02ed2c0e0dfde9 P 465dae58ab594882ba704e044efba81d: Deleting consensus metadata
I20260928 12:26:30.771262 14738 catalog_manager.cc:4841] TS 465dae58ab594882ba704e044efba81d (127.7.19.66:44315): tablet c91e05ad660c4f419e02ed2c0e0dfde9 (table test-table [id=6a0bb8e654c6416488fa734dac552c67]) successfully deleted
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:26:30.785615 15302 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:30.779260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.7.19.126:35971)
0928 12:26:30.779326 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.779328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.779477 (+ 149us) server_negotiation.cc:407] Connection header received
0928 12:26:30.779549 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.779550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.779563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.779611 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.780318 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.780920 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.782034 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.782232 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.782290 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.785536 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39254: BlockingRecv error: recv got EOF from 127.0.0.1:39254 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 12:26:30 dist-test-slave-cb7x krb5kdc[14706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM
I20260928 12:26:30.794452 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:30.786356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.7.19.126:35971)
0928 12:26:30.786393 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:30.786395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:30.786615 (+ 220us) server_negotiation.cc:407] Connection header received
0928 12:26:30.786645 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:30.786646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:30.786658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:30.786719 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:30.787138 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.787887 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.788622 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:30.788957 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:30.789027 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:30.791722 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:30.791724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:30.791724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:30.791732 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:30.793402 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.793717 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.793719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.793719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.793778 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:30.793969 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:30.793970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:30.793970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:30.794102 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:30.794203 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:30.794276 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 12:26:30.794352 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:26:30.796280 14750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:30.796465 14750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:30.797881 14889 tablet_service.cc:1467] Processing CreateTablet for tablet 7654a033e5594e08a6b349850c7fc768 (DEFAULT_TABLE table=test-table [id=bdeef771c98846a1aa5d9ae6d24cdffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.797881 15029 tablet_service.cc:1467] Processing CreateTablet for tablet 7654a033e5594e08a6b349850c7fc768 (DEFAULT_TABLE table=test-table [id=bdeef771c98846a1aa5d9ae6d24cdffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.798038 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7654a033e5594e08a6b349850c7fc768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.798038 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7654a033e5594e08a6b349850c7fc768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.799268 15268 tablet_bootstrap.cc:492] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Bootstrap starting.
I20260928 12:26:30.799294 15264 tablet_bootstrap.cc:492] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Bootstrap starting.
I20260928 12:26:30.799612 15268 tablet_bootstrap.cc:654] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.799695 15264 tablet_bootstrap.cc:654] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.800051 15169 tablet_service.cc:1467] Processing CreateTablet for tablet 7654a033e5594e08a6b349850c7fc768 (DEFAULT_TABLE table=test-table [id=bdeef771c98846a1aa5d9ae6d24cdffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:30.800163 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7654a033e5594e08a6b349850c7fc768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:30.800266 15268 tablet_bootstrap.cc:492] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: No bootstrap required, opened a new log
I20260928 12:26:30.800298 15268 ts_tablet_manager.cc:1397] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:30.800384 15268 raft_consensus.cc:348] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.800465 15268 raft_consensus.cc:374] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.800484 15268 raft_consensus.cc:729] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465dae58ab594882ba704e044efba81d, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.800546 15268 consensus_queue.cc:260] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.800637 15268 ts_tablet_manager.cc:1428] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.801414 15263 tablet_bootstrap.cc:492] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Bootstrap starting.
I20260928 12:26:30.801793 15263 tablet_bootstrap.cc:654] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:30.802508 15263 tablet_bootstrap.cc:492] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: No bootstrap required, opened a new log
I20260928 12:26:30.802527 15264 tablet_bootstrap.cc:492] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: No bootstrap required, opened a new log
I20260928 12:26:30.802572 15264 ts_tablet_manager.cc:1397] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:26:30.802574 15263 ts_tablet_manager.cc:1397] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:30.802668 15264 raft_consensus.cc:348] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.802755 15264 raft_consensus.cc:374] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.802732 15263 raft_consensus.cc:348] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.802810 15264 raft_consensus.cc:729] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.802819 15263 raft_consensus.cc:374] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:30.802841 15263 raft_consensus.cc:729] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e7fc013c32483682dce7ffc46e40df, State: Initialized, Role: FOLLOWER
I20260928 12:26:30.802858 15264 consensus_queue.cc:260] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.802886 15263 consensus_queue.cc:260] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:30.802956 15264 ts_tablet_manager.cc:1428] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:30.803030 15263 ts_tablet_manager.cc:1428] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:31.094573 15270 raft_consensus.cc:482] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:31.094687 15270 raft_consensus.cc:504] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.095141 15270 leader_election.cc:288] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913), 465dae58ab594882ba704e044efba81d (127.7.19.66:44315)
I20260928 12:26:31.098167 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.095388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:50363 (local address 127.7.19.66:44315)
0928 12:26:31.095493 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:31.095495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:31.095557 (+ 62us) server_negotiation.cc:407] Connection header received
0928 12:26:31.095622 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:31.095623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:31.095633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:31.095718 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:31.095853 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.096390 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.097451 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.097817 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.097898 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:31.097910 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:31.098062 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:26:31.098102 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":45,"threads_started":1}
I20260928 12:26:31.098174 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.095343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:44315 (local address 127.7.19.67:50363)
0928 12:26:31.095485 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:31.095496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:31.095560 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:31.095718 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:31.095719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:31.095721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:31.095781 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:31.095783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.096474 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.096475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:31.097350 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:31.097352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.097899 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.097900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:31.097977 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:31.097980 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:31.098044 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:26:31.098087 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260928 12:26:31.098232 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.095450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:48865 (local address 127.7.19.65:38913)
0928 12:26:31.095497 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:31.095499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:31.095587 (+ 88us) server_negotiation.cc:407] Connection header received
0928 12:26:31.095646 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:31.095647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:31.095656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:31.095702 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:31.095962 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.096439 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.097493 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.097874 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.097955 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:31.097965 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:31.098123 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 12:26:31.098159 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:26:31.098302 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.095390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38913 (local address 127.7.19.67:48865)
0928 12:26:31.095498 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:31.095508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:31.095572 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:31.095714 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:31.095715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:31.095717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:31.095810 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:31.095812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.096547 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.096548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:31.097395 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:31.097398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.097967 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.097968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:31.098039 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:31.098045 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:26:31.098118 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:26:31.098159 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":21,"thread_start_us":52,"threads_started":1}
I20260928 12:26:31.098692 14908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7654a033e5594e08a6b349850c7fc768" candidate_uuid: "a9e7fc013c32483682dce7ffc46e40df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc41ff9712a42e2aeb73feb801d6369" is_pre_election: true
I20260928 12:26:31.098801 14908 raft_consensus.cc:2454] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9e7fc013c32483682dce7ffc46e40df in term 0.
I20260928 12:26:31.098764 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7654a033e5594e08a6b349850c7fc768" candidate_uuid: "a9e7fc013c32483682dce7ffc46e40df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d" is_pre_election: true
I20260928 12:26:31.098879 15049 raft_consensus.cc:2454] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9e7fc013c32483682dce7ffc46e40df in term 0.
I20260928 12:26:31.099015 15120 leader_election.cc:302] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:31.099161 15270 raft_consensus.cc:2781] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:31.099197 15270 raft_consensus.cc:482] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:31.099221 15270 raft_consensus.cc:3037] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:31.100029 15270 raft_consensus.cc:504] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.100198 15270 leader_election.cc:288] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [CANDIDATE]: Term 1 election: Requested vote from peers 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913), 465dae58ab594882ba704e044efba81d (127.7.19.66:44315)
I20260928 12:26:31.100312 14908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7654a033e5594e08a6b349850c7fc768" candidate_uuid: "a9e7fc013c32483682dce7ffc46e40df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fc41ff9712a42e2aeb73feb801d6369"
I20260928 12:26:31.100359 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7654a033e5594e08a6b349850c7fc768" candidate_uuid: "a9e7fc013c32483682dce7ffc46e40df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d"
I20260928 12:26:31.100399 14908 raft_consensus.cc:3037] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:31.100425 15049 raft_consensus.cc:3037] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:31.101197 14908 raft_consensus.cc:2454] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e7fc013c32483682dce7ffc46e40df in term 1.
I20260928 12:26:31.101197 15049 raft_consensus.cc:2454] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e7fc013c32483682dce7ffc46e40df in term 1.
I20260928 12:26:31.101393 15123 leader_election.cc:302] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465dae58ab594882ba704e044efba81d, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:31.101526 15270 raft_consensus.cc:2781] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:31.101660 15270 raft_consensus.cc:686] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 1 LEADER]: Becoming Leader. State: Replica: a9e7fc013c32483682dce7ffc46e40df, State: Running, Role: LEADER
I20260928 12:26:31.101797 15270 consensus_queue.cc:237] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.102624 14749 catalog_manager.cc:5495] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df reported cstate change: term changed from 0 to 1, leader changed from <none> to a9e7fc013c32483682dce7ffc46e40df (127.7.19.67). New cstate: current_term: 1 leader_uuid: "a9e7fc013c32483682dce7ffc46e40df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } health_report { overall_health: HEALTHY } } }
I20260928 12:26:31.154131 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.151670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.7.19.67:44317)
0928 12:26:31.151785 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:31.151787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:31.151795 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:31.151829 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:31.151830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:31.151848 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:31.151920 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:31.152106 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.152613 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.153360 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.153561 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.153616 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:31.153745 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:31.153846 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:31.153912 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:31.153959 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:26:31.153975 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260928 12:26:31.155230 15049 raft_consensus.cc:1261] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Refusing update from remote peer a9e7fc013c32483682dce7ffc46e40df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:31.155236 14908 raft_consensus.cc:1261] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Refusing update from remote peer a9e7fc013c32483682dce7ffc46e40df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:31.155527 15270 consensus_queue.cc:1035] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [LEADER]: Connected to new peer: Peer: permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:31.155651 15337 consensus_queue.cc:1035] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:31.158715 14750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:31.158808 14750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:31.159411 14750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Sending DeleteTablet for 3 replicas of tablet 7654a033e5594e08a6b349850c7fc768
I20260928 12:26:31.159642 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet 7654a033e5594e08a6b349850c7fc768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:31 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:32860
I20260928 12:26:31.159660 15029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7654a033e5594e08a6b349850c7fc768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:31 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260928 12:26:31.159744 15286 tablet_replica.cc:330] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: stopping tablet replica
I20260928 12:26:31.159736 14889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7654a033e5594e08a6b349850c7fc768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:31 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:54062
I20260928 12:26:31.159790 15286 raft_consensus.cc:2229] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:31.159814 15287 tablet_replica.cc:330] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: stopping tablet replica
I20260928 12:26:31.159830 15286 raft_consensus.cc:2258] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:31.159884 15287 raft_consensus.cc:2229] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:31.159864 14750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:26:31.159955 15287 raft_consensus.cc:2258] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:26:31.159996 14750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:31.160025 15288 tablet_replica.cc:330] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: stopping tablet replica
I20260928 12:26:31.160075 15288 raft_consensus.cc:2229] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:31.160121 15288 raft_consensus.cc:2258] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:31.160395 15288 ts_tablet_manager.cc:1905] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:31.160459 15286 ts_tablet_manager.cc:1905] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:31.160626 15287 ts_tablet_manager.cc:1905] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:31.162060 15287 ts_tablet_manager.cc:1918] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:31.162087 15286 ts_tablet_manager.cc:1918] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:31.162092 15288 ts_tablet_manager.cc:1918] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:31.162130 15286 log.cc:1196] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/7654a033e5594e08a6b349850c7fc768
I20260928 12:26:31.162130 15287 log.cc:1196] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/7654a033e5594e08a6b349850c7fc768
I20260928 12:26:31.162142 15288 log.cc:1196] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/7654a033e5594e08a6b349850c7fc768
I20260928 12:26:31.162353 15287 ts_tablet_manager.cc:1939] T 7654a033e5594e08a6b349850c7fc768 P a9e7fc013c32483682dce7ffc46e40df: Deleting consensus metadata
I20260928 12:26:31.162353 15286 ts_tablet_manager.cc:1939] T 7654a033e5594e08a6b349850c7fc768 P 465dae58ab594882ba704e044efba81d: Deleting consensus metadata
I20260928 12:26:31.162382 15288 ts_tablet_manager.cc:1939] T 7654a033e5594e08a6b349850c7fc768 P 2fc41ff9712a42e2aeb73feb801d6369: Deleting consensus metadata
I20260928 12:26:31.162823 14735 catalog_manager.cc:4841] TS 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913): tablet 7654a033e5594e08a6b349850c7fc768 (table test-table [id=bdeef771c98846a1aa5d9ae6d24cdffa]) successfully deleted
I20260928 12:26:31.163005 14738 catalog_manager.cc:4841] TS 465dae58ab594882ba704e044efba81d (127.7.19.66:44315): tablet 7654a033e5594e08a6b349850c7fc768 (table test-table [id=bdeef771c98846a1aa5d9ae6d24cdffa]) successfully deleted
I20260928 12:26:31.163280 14889 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e04d727964b3aafcb111b24bfe5a3 (DEFAULT_TABLE table=test-table [id=17ea740251bb4aa397be648ab9222ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:31.163308 15029 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e04d727964b3aafcb111b24bfe5a3 (DEFAULT_TABLE table=test-table [id=17ea740251bb4aa397be648ab9222ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:31.163380 14889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e04d727964b3aafcb111b24bfe5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:31.163404 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e04d727964b3aafcb111b24bfe5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:31.163552 14736 catalog_manager.cc:4841] TS a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317): tablet 7654a033e5594e08a6b349850c7fc768 (table test-table [id=bdeef771c98846a1aa5d9ae6d24cdffa]) successfully deleted
I20260928 12:26:31.164263 15268 tablet_bootstrap.cc:492] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Bootstrap starting.
I20260928 12:26:31.164472 15169 tablet_service.cc:1467] Processing CreateTablet for tablet 4a0e04d727964b3aafcb111b24bfe5a3 (DEFAULT_TABLE table=test-table [id=17ea740251bb4aa397be648ab9222ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:31.164575 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a0e04d727964b3aafcb111b24bfe5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:31.164584 15268 tablet_bootstrap.cc:654] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:31.165096 15268 tablet_bootstrap.cc:492] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: No bootstrap required, opened a new log
I20260928 12:26:31.165154 15268 ts_tablet_manager.cc:1397] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:31.165254 15268 raft_consensus.cc:348] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.165338 15268 raft_consensus.cc:374] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:31.165364 15268 raft_consensus.cc:729] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465dae58ab594882ba704e044efba81d, State: Initialized, Role: FOLLOWER
I20260928 12:26:31.165414 15268 consensus_queue.cc:260] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.165500 15268 ts_tablet_manager.cc:1428] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:31.165752 15264 tablet_bootstrap.cc:492] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369: Bootstrap starting.
I20260928 12:26:31.166074 15264 tablet_bootstrap.cc:654] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:31.166298 15263 tablet_bootstrap.cc:492] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Bootstrap starting.
I20260928 12:26:31.166599 15264 tablet_bootstrap.cc:492] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369: No bootstrap required, opened a new log
I20260928 12:26:31.166661 15264 ts_tablet_manager.cc:1397] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:31.166666 15263 tablet_bootstrap.cc:654] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:31.166764 15264 raft_consensus.cc:348] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.166844 15264 raft_consensus.cc:374] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:31.166872 15264 raft_consensus.cc:729] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Initialized, Role: FOLLOWER
I20260928 12:26:31.166939 15264 consensus_queue.cc:260] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.167037 15264 ts_tablet_manager.cc:1428] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:31.167439 15263 tablet_bootstrap.cc:492] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: No bootstrap required, opened a new log
I20260928 12:26:31.167495 15263 ts_tablet_manager.cc:1397] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:31.167614 15263 raft_consensus.cc:348] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.167680 15263 raft_consensus.cc:374] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:31.167701 15263 raft_consensus.cc:729] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e7fc013c32483682dce7ffc46e40df, State: Initialized, Role: FOLLOWER
I20260928 12:26:31.167740 15263 consensus_queue.cc:260] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.167825 15263 ts_tablet_manager.cc:1428] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:31.299777 15294 raft_consensus.cc:482] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:31.299888 15294 raft_consensus.cc:504] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.300088 15294 leader_election.cc:288] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 465dae58ab594882ba704e044efba81d (127.7.19.66:44315), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:31.300348 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e04d727964b3aafcb111b24bfe5a3" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df" is_pre_election: true
I20260928 12:26:31.300297 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e04d727964b3aafcb111b24bfe5a3" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d" is_pre_election: true
I20260928 12:26:31.300428 15189 raft_consensus.cc:2454] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 0.
I20260928 12:26:31.300452 15049 raft_consensus.cc:2454] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 0.
I20260928 12:26:31.300603 14841 leader_election.cc:302] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:31.300721 15294 raft_consensus.cc:2781] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:31.300767 15294 raft_consensus.cc:482] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:31.300782 15294 raft_consensus.cc:3037] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:31.301537 15294 raft_consensus.cc:504] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.301708 15294 leader_election.cc:288] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 election: Requested vote from peers 465dae58ab594882ba704e044efba81d (127.7.19.66:44315), a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317)
I20260928 12:26:31.301844 15049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e04d727964b3aafcb111b24bfe5a3" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "465dae58ab594882ba704e044efba81d"
I20260928 12:26:31.301872 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a0e04d727964b3aafcb111b24bfe5a3" candidate_uuid: "2fc41ff9712a42e2aeb73feb801d6369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e7fc013c32483682dce7ffc46e40df"
I20260928 12:26:31.301937 15049 raft_consensus.cc:3037] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:31.301944 15189 raft_consensus.cc:3037] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:31.302532 15049 raft_consensus.cc:2454] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 1.
I20260928 12:26:31.302537 15189 raft_consensus.cc:2454] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc41ff9712a42e2aeb73feb801d6369 in term 1.
I20260928 12:26:31.302752 14841 leader_election.cc:302] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc41ff9712a42e2aeb73feb801d6369, a9e7fc013c32483682dce7ffc46e40df; no voters:
I20260928 12:26:31.302862 15294 raft_consensus.cc:2781] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:31.302970 15294 raft_consensus.cc:686] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 LEADER]: Becoming Leader. State: Replica: 2fc41ff9712a42e2aeb73feb801d6369, State: Running, Role: LEADER
I20260928 12:26:31.303026 15294 consensus_queue.cc:237] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } }
I20260928 12:26:31.303575 14750 catalog_manager.cc:5495] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "2fc41ff9712a42e2aeb73feb801d6369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc41ff9712a42e2aeb73feb801d6369" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 } health_report { overall_health: UNKNOWN } } }
I20260928 12:26:31.315521 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.313346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.7.19.65:38913)
0928 12:26:31.313387 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:31.313389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:31.313398 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:31.313468 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:31.313470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:31.313479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:31.313531 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:31.313639 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.314117 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.314840 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:31.315047 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:31.315121 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:31.315264 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:31.315308 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:31.315365 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:31.315425 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 12:26:31.315446 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:26:31.317633 15189 raft_consensus.cc:1261] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Refusing update from remote peer 2fc41ff9712a42e2aeb73feb801d6369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:31.317718 15049 raft_consensus.cc:1261] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Refusing update from remote peer 2fc41ff9712a42e2aeb73feb801d6369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:31.317839 15348 consensus_queue.cc:1035] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e7fc013c32483682dce7ffc46e40df" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:31.317941 15350 consensus_queue.cc:1035] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "465dae58ab594882ba704e044efba81d" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:31.327144 14749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:31.327232 14749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:31.327797 14749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5cd341439424d5a88f3fa1059998273: Sending DeleteTablet for 3 replicas of tablet 4a0e04d727964b3aafcb111b24bfe5a3
I20260928 12:26:31.327987 14889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0e04d727964b3aafcb111b24bfe5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:31 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:54062
I20260928 12:26:31.328025 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0e04d727964b3aafcb111b24bfe5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:31 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:32860
I20260928 12:26:31.328073 15288 tablet_replica.cc:330] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369: stopping tablet replica
I20260928 12:26:31.328063 15029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a0e04d727964b3aafcb111b24bfe5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:31 UTC) from {username='oryx', principal='oryx/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260928 12:26:31.328097 15287 tablet_replica.cc:330] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: stopping tablet replica
I20260928 12:26:31.328122 15288 raft_consensus.cc:2229] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:31.328136 15287 raft_consensus.cc:2229] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:31.328150 15286 tablet_replica.cc:330] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: stopping tablet replica
I20260928 12:26:31.328167 15287 raft_consensus.cc:2258] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:31.328176 15288 raft_consensus.cc:2258] T 4a0e04d727964b3aafcb111b24bfe5a3 P 2fc41ff9712a42e2aeb73feb801d6369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:31.328199 15286 raft_consensus.cc:2229] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:31.328231 15286 raft_consensus.cc:2258] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:31.328393 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 14824
I20260928 12:26:31.330114 15286 ts_tablet_manager.cc:1905] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:31.330565 15287 ts_tablet_manager.cc:1905] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:31.331511 15286 ts_tablet_manager.cc:1918] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:31.331583 15286 log.cc:1196] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/4a0e04d727964b3aafcb111b24bfe5a3
I20260928 12:26:31.331794 15286 ts_tablet_manager.cc:1939] T 4a0e04d727964b3aafcb111b24bfe5a3 P 465dae58ab594882ba704e044efba81d: Deleting consensus metadata
I20260928 12:26:31.332134 14738 catalog_manager.cc:4841] TS 465dae58ab594882ba704e044efba81d (127.7.19.66:44315): tablet 4a0e04d727964b3aafcb111b24bfe5a3 (table test-table [id=17ea740251bb4aa397be648ab9222ec8]) successfully deleted
I20260928 12:26:31.332274 15287 ts_tablet_manager.cc:1918] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 12:26:31.332335 15287 log.cc:1196] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/4a0e04d727964b3aafcb111b24bfe5a3
I20260928 12:26:31.332543 15287 ts_tablet_manager.cc:1939] T 4a0e04d727964b3aafcb111b24bfe5a3 P a9e7fc013c32483682dce7ffc46e40df: Deleting consensus metadata
I20260928 12:26:31.332890 14736 catalog_manager.cc:4841] TS a9e7fc013c32483682dce7ffc46e40df (127.7.19.67:44317): tablet 4a0e04d727964b3aafcb111b24bfe5a3 (table test-table [id=17ea740251bb4aa397be648ab9222ec8]) successfully deleted
W20260928 12:26:31.335352 14735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:38913 (error 108)
I20260928 12:26:31.335587 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 14964
I20260928 12:26:31.336225 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:31.335669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38913 (local address 127.0.0.1:54106)
0928 12:26:31.336123 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:31.336171 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":39,"threads_started":1}
W20260928 12:26:31.336351 14735 catalog_manager.cc:4584] TS 2fc41ff9712a42e2aeb73feb801d6369 (127.7.19.65:38913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a0e04d727964b3aafcb111b24bfe5a3: Network error: Client connection negotiation failed: client connection to 127.7.19.65:38913: connect: Connection refused (error 111)
I20260928 12:26:31.342070 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15104
I20260928 12:26:31.347661 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 14719
2026-09-28T12:26:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3157 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:31 dist-test-slave-cb7x krb5kdc[15359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:31 dist-test-slave-cb7x krb5kdc[15359](info): set up 2 sockets
Sep 28 12:26:31 dist-test-slave-cb7x krb5kdc[15359](info): commencing operation
krb5kdc: starting...
W20260928 12:26:32.377316 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:32 dist-test-slave-cb7x krb5kdc[15359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:32.492202 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46791
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:32.571801 15372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:32.571996 15372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:32.572021 15372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:32.573122 15372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:32.573155 15372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:32.573169 15372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:32.573181 15372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:32.573194 15372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:32.574609 15372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:32.574790 15372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:32.574949 15372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:32.579921 15372 server_base.cc:1034] running on GCE node
W20260928 12:26:32.579900 15378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:32.579938 15377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:32.580127 15380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:32.580436 15372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:32.580719 15372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:32.581903 15372 hybrid_clock.cc:648] HybridClock initialized: now 1790598392581890 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T12:26:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1242 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:32 dist-test-slave-cb7x krb5kdc[15385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:32 dist-test-slave-cb7x krb5kdc[15385](info): set up 2 sockets
Sep 28 12:26:32 dist-test-slave-cb7x krb5kdc[15385](info): commencing operation
krb5kdc: starting...
W20260928 12:26:33.641319 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:33 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:33.760430 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34961
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:33.849431 15398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:33.849637 15398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:33.849661 15398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:33.850981 15398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:33.851030 15398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:33.851046 15398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:33.851101 15398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:33.851126 15398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:33.852837 15398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:33.853084 15398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:33.853359 15398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:33.858232 15404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:33.858347 15398 server_base.cc:1034] running on GCE node
W20260928 12:26:33.858222 15403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:33.858523 15406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:33.858783 15398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:33.859078 15398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:33.860252 15398 hybrid_clock.cc:648] HybridClock initialized: now 1790598393860233 us; error 35 us; skew 500 ppm
Sep 28 12:26:33 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598393, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:33.862764 15398 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:33.863212 15398 webserver.cc:460] Webserver started at http://127.7.19.126:39009/ using document root <none> and password file <none>
I20260928 12:26:33.863451 15398 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:33.863517 15398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:33.863663 15398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:33.864728 15398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "6a292b2fcd11419697fb054f5c79fcf6"
format_stamp: "Formatted at 2026-09-28 12:26:33 on dist-test-slave-cb7x"
server_key: "871a98dbf2e5b762c937b47a68631d0a"
server_key_iv: "ab6420c9023fd4759417f93332d4453b"
server_key_version: "encryptionkey@0"
I20260928 12:26:33.865120 15398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "6a292b2fcd11419697fb054f5c79fcf6"
format_stamp: "Formatted at 2026-09-28 12:26:33 on dist-test-slave-cb7x"
server_key: "871a98dbf2e5b762c937b47a68631d0a"
server_key_iv: "ab6420c9023fd4759417f93332d4453b"
server_key_version: "encryptionkey@0"
I20260928 12:26:33.866493 15398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:33.867247 15413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:33.867460 15398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:33.867556 15398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "6a292b2fcd11419697fb054f5c79fcf6"
format_stamp: "Formatted at 2026-09-28 12:26:33 on dist-test-slave-cb7x"
server_key: "871a98dbf2e5b762c937b47a68631d0a"
server_key_iv: "ab6420c9023fd4759417f93332d4453b"
server_key_version: "encryptionkey@0"
I20260928 12:26:33.867635 15398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:33.892210 15398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:33.894145 15398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:33.894313 15398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:33.898427 15398 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:34961
I20260928 12:26:33.898463 15465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:34961 every 8 connection(s)
I20260928 12:26:33.898814 15398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:33.899673 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:33.902025 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Bootstrap starting.
I20260928 12:26:33.902536 15466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:33.902771 15466 log.cc:824] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:33.903419 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: No bootstrap required, opened a new log
I20260928 12:26:33.905018 15466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:33.905154 15466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:33.905177 15466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a292b2fcd11419697fb054f5c79fcf6, State: Initialized, Role: FOLLOWER
I20260928 12:26:33.905323 15466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:33.905421 15466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:33.905447 15466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:33.905500 15466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:33.905506 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15398
I20260928 12:26:33.905620 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:33.905807 7245 external_mini_cluster.cc:1428] Setting key ad30b2f1d8cf9d48e31d9e5042493720
I20260928 12:26:33.906193 15466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:33.906361 15466 leader_election.cc:302] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a292b2fcd11419697fb054f5c79fcf6; no voters:
I20260928 12:26:33.906548 15466 leader_election.cc:288] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:33.906746 15471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:33.906795 15466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:33.906898 15471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 LEADER]: Becoming Leader. State: Replica: 6a292b2fcd11419697fb054f5c79fcf6, State: Running, Role: LEADER
I20260928 12:26:33.906992 15471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:33.907310 15473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a292b2fcd11419697fb054f5c79fcf6. Latest consensus state: current_term: 1 leader_uuid: "6a292b2fcd11419697fb054f5c79fcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } } }
I20260928 12:26:33.907409 15473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:33.907630 15480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:33.908079 15480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:33.908191 15472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a292b2fcd11419697fb054f5c79fcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } } }
I20260928 12:26:33.908358 15472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:33.909686 15480 catalog_manager.cc:1347] Generated new cluster ID: 0e46681bc1984388a8363263df7a2f49
I20260928 12:26:33.909735 15480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:26:33 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:33.918107 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:33.906315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.7.19.126:34961)
0928 12:26:33.906595 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:33.906606 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:33.906647 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:26:33.906774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:33.906781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:33.906844 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:33.906994 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:33.907838 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:33.909269 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:33.910318 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:33.910901 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:33.910998 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:33.913774 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:33.913778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:33.913778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:33.913790 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:33.916424 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:33.916828 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:33.916832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:33.916832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:33.916952 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:33.917152 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:33.917153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:33.917153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:33.917483 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:33.917661 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:33.917731 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 12:26:33.917883 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":127,"threads_started":1}
I20260928 12:26:33.922026 15480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:33.922490 15480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:33.927546 15480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Generated new TSK 0
I20260928 12:26:33.927727 15480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:33.955742 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34961
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:34.036257 15492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.036489 15492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.036527 15492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.037744 15492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.037811 15492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.037858 15492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:34.039283 15492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.039548 15492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.039774 15492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.044718 15497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.044747 15492 server_base.cc:1034] running on GCE node
W20260928 12:26:34.044756 15500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.044895 15498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.045118 15492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.045372 15492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.046559 15492 hybrid_clock.cc:648] HybridClock initialized: now 1790598394046541 us; error 51 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.049136 15492 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:34.049567 15492 webserver.cc:460] Webserver started at http://127.7.19.65:43171/ using document root <none> and password file <none>
I20260928 12:26:34.049795 15492 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.049885 15492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.050032 15492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:34.051066 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "e570142e26024ba581485daa322bcf00"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "aecb58ac6012c1b74943974add3f2978"
server_key_iv: "4c0c96885c622d2d430249650a7438ef"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.051430 15492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "e570142e26024ba581485daa322bcf00"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "aecb58ac6012c1b74943974add3f2978"
server_key_iv: "4c0c96885c622d2d430249650a7438ef"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.052781 15492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:34.053511 15507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.053681 15492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:34.053778 15492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "e570142e26024ba581485daa322bcf00"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "aecb58ac6012c1b74943974add3f2978"
server_key_iv: "4c0c96885c622d2d430249650a7438ef"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.053861 15492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.066843 15492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.068984 15492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.069144 15492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.069373 15492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:34.069680 15492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:34.069736 15492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.069772 15492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:34.069799 15492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.075683 15492 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:35901
I20260928 12:26:34.075719 15620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:35901 every 8 connection(s)
I20260928 12:26:34.076045 15492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:34.080353 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15492
I20260928 12:26:34.080476 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:34.080736 7245 external_mini_cluster.cc:1428] Setting key 84e172864a38eb9d6369bd60f7150352
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:34.085764 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.076796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51357 (local address 127.7.19.126:34961)
0928 12:26:34.076856 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.076858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.077050 (+ 192us) server_negotiation.cc:407] Connection header received
0928 12:26:34.077603 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.077604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.077620 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.077675 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.078598 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.079111 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.080102 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.080338 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.080427 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.082781 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:34.082782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:34.082783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:34.082792 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:34.084131 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:34.084622 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:34.084623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:34.084623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:34.084755 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:34.085085 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:34.085086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:34.085087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:34.085231 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:34.085290 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:34.085629 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 12:26:34.085702 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:26:34.086043 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.076784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:51357)
0928 12:26:34.076935 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.076968 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.077498 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.077751 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.077754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.078298 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.078495 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.078500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.079218 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.079219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.080014 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.080016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.080437 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.080439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.080610 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.081212 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.081221 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.082682 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:34.084206 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:34.084213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:34.084216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:34.084530 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:34.084835 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:34.084836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:34.084837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:34.084989 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:34.085304 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:34.085311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:34.085477 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 12:26:34.085602 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 12:26:34.085776 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260928 12:26:34.086592 15621 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34961
I20260928 12:26:34.086700 15621 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:34.086916 15621 heartbeater.cc:511] Master 127.7.19.126:34961 requested a full tablet report, sending...
I20260928 12:26:34.087437 15430 ts_manager.cc:194] Registered new tserver with Master: e570142e26024ba581485daa322bcf00 (127.7.19.65:35901)
I20260928 12:26:34.088063 15430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:51357
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:34.107263 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34961
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:34.187510 15626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.187701 15626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.187723 15626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.188831 15626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.188868 15626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.188894 15626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:34.190207 15626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.190382 15626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.190544 15626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.195434 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.195453 15632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.195571 15626 server_base.cc:1034] running on GCE node
W20260928 12:26:34.195500 15631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.195897 15626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.196130 15626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.197289 15626 hybrid_clock.cc:648] HybridClock initialized: now 1790598394197270 us; error 51 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.199719 15626 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:34.200148 15626 webserver.cc:460] Webserver started at http://127.7.19.66:33563/ using document root <none> and password file <none>
I20260928 12:26:34.200323 15626 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.200368 15626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.200455 15626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:34.201562 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "7ffbcf5cbf884f65b1eb25d7731c79a4"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "bd34335934eeb3faa6fbe3dbe484901b"
server_key_iv: "da667383d0ea4632fa4740668d293760"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.201927 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "7ffbcf5cbf884f65b1eb25d7731c79a4"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "bd34335934eeb3faa6fbe3dbe484901b"
server_key_iv: "da667383d0ea4632fa4740668d293760"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.203229 15626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:26:34.203996 15641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.204183 15626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:34.204283 15626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "7ffbcf5cbf884f65b1eb25d7731c79a4"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "bd34335934eeb3faa6fbe3dbe484901b"
server_key_iv: "da667383d0ea4632fa4740668d293760"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.204362 15626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.213826 15626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.215685 15626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.215834 15626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.216030 15626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:34.216311 15626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:34.216374 15626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.216416 15626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:34.216441 15626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.222124 15626 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:44335
I20260928 12:26:34.222167 15754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:44335 every 8 connection(s)
I20260928 12:26:34.222479 15626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:34.231649 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.223159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:58839 (local address 127.7.19.126:34961)
0928 12:26:34.223222 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.223224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.223413 (+ 189us) server_negotiation.cc:407] Connection header received
0928 12:26:34.223934 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.223936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.223947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.224000 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.224951 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.225469 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.226483 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.226690 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.226767 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.228812 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:34.228813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:34.228814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:34.228820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:34.230247 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:34.230678 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:34.230679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:34.230679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:34.230732 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:34.230999 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:34.231001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:34.231001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:34.231117 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:34.231185 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:34.231508 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 12:26:34.231585 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:34.231807 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.223146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:58839)
0928 12:26:34.223294 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.223327 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.223829 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.224086 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.224090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.224638 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.224848 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.224853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.225574 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.225575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.226384 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.226386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.226773 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.226775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.226896 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.227340 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.227348 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.228697 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:34.230340 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:34.230346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:34.230347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:34.230601 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:34.230812 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:34.230813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:34.230813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:34.230919 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:34.231213 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:34.231218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:34.231350 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 12:26:34.231459 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 12:26:34.231588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260928 12:26:34.231932 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15626
I20260928 12:26:34.232048 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:34.232235 7245 external_mini_cluster.cc:1428] Setting key 971e19731ec499d08cd1c9f1ceaeba31
I20260928 12:26:34.232388 15755 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34961
I20260928 12:26:34.232501 15755 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:34.232702 15755 heartbeater.cc:511] Master 127.7.19.126:34961 requested a full tablet report, sending...
I20260928 12:26:34.233139 15430 ts_manager.cc:194] Registered new tserver with Master: 7ffbcf5cbf884f65b1eb25d7731c79a4 (127.7.19.66:44335)
I20260928 12:26:34.233688 15430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:58839
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:34.259871 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34961
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:34.340472 15760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.340728 15760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.340768 15760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.341972 15760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.342041 15760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.342083 15760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:34.343487 15760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.343743 15760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.343967 15760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.348939 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.348929 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.348956 15760 server_base.cc:1034] running on GCE node
W20260928 12:26:34.348925 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.349385 15760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.349638 15760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.350777 15760 hybrid_clock.cc:648] HybridClock initialized: now 1790598394350758 us; error 40 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.353359 15760 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:34.353782 15760 webserver.cc:460] Webserver started at http://127.7.19.67:46383/ using document root <none> and password file <none>
I20260928 12:26:34.354048 15760 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.354101 15760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.354240 15760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:34.355285 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "357ace810fcf41c19d3f9b1fdfb54813"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "3a6edde4f266189459b4774cc6e389be"
server_key_iv: "81b37a4fff2e4162a461e3dab3b3c00a"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.355651 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "357ace810fcf41c19d3f9b1fdfb54813"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "3a6edde4f266189459b4774cc6e389be"
server_key_iv: "81b37a4fff2e4162a461e3dab3b3c00a"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.357100 15760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:34.357926 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.358258 15760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:34.358333 15760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "357ace810fcf41c19d3f9b1fdfb54813"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "3a6edde4f266189459b4774cc6e389be"
server_key_iv: "81b37a4fff2e4162a461e3dab3b3c00a"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.358421 15760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.371025 15760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.372892 15760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.373041 15760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.373260 15760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:34.373548 15760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:34.373579 15760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.373627 15760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:34.373656 15760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.379443 15760 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:44249
I20260928 12:26:34.379520 15888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:44249 every 8 connection(s)
I20260928 12:26:34.379813 15760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:34.384512 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15760
I20260928 12:26:34.384611 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:34.384847 7245 external_mini_cluster.cc:1428] Setting key 1044f7ced84c32be739e5d66ecc9a394
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:34.388948 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.380555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:45141 (local address 127.7.19.126:34961)
0928 12:26:34.380625 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.380628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.380898 (+ 270us) server_negotiation.cc:407] Connection header received
0928 12:26:34.381354 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.381355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.381365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.381429 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.382310 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.382795 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.383752 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.383938 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.383987 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.386260 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:34.386262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:34.386262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:34.386269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:34.387566 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:34.387978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:34.387979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:34.387979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:34.388043 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:34.388288 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:34.388289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:34.388290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:34.388411 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:34.388487 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:34.388774 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 12:26:34.388850 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:34.389137 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.380541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:45141)
0928 12:26:34.380777 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.380814 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.381276 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.381484 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.381487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.382047 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.382235 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.382239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.382871 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.382872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.383681 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.383683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.384006 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.384007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.384111 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.384733 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.384743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.386151 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:34.387654 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:34.387660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:34.387662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:34.387885 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:34.388108 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:34.388109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:34.388109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:34.388220 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:34.388478 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:34.388483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:34.388622 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 12:26:34.388754 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 12:26:34.388912 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
I20260928 12:26:34.389640 15889 heartbeater.cc:347] Connected to a master server at 127.7.19.126:34961
I20260928 12:26:34.389770 15889 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:34.389973 15889 heartbeater.cc:511] Master 127.7.19.126:34961 requested a full tablet report, sending...
I20260928 12:26:34.390334 15430 ts_manager.cc:194] Registered new tserver with Master: 357ace810fcf41c19d3f9b1fdfb54813 (127.7.19.67:44249)
I20260928 12:26:34.390714 15430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:45141
I20260928 12:26:34.396412 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab.
Entry for principal oryx/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab.
WARNING: no policy specified for oryx/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab.
Entry for principal oryx/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab.
WARNING: no policy specified for oryx/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab.
Entry for principal oryx/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab.
I20260928 12:26:34.465358 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15492
I20260928 12:26:34.469844 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15626
I20260928 12:26:34.474121 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15760
I20260928 12:26:34.478348 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15398
I20260928 12:26:34.483608 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34961
--webserver_interface=127.7.19.126
--webserver_port=39009
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:34.563910 15898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.564112 15898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.564134 15898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.565292 15898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:34.565332 15898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.565346 15898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.565358 15898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:34.565371 15898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:34.566748 15898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=39009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.566936 15898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.567106 15898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.572026 15906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.572026 15903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.572059 15904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.572072 15898 server_base.cc:1034] running on GCE node
I20260928 12:26:34.572477 15898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.572778 15898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.573954 15898 hybrid_clock.cc:648] HybridClock initialized: now 1790598394573948 us; error 36 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.584487 15898 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:34.585017 15898 webserver.cc:460] Webserver started at http://127.7.19.126:39009/ using document root <none> and password file <none>
I20260928 12:26:34.585233 15898 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.585287 15898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.586707 15898 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:34.587491 15913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.587761 15898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:34.587872 15898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "6a292b2fcd11419697fb054f5c79fcf6"
format_stamp: "Formatted at 2026-09-28 12:26:33 on dist-test-slave-cb7x"
server_key: "871a98dbf2e5b762c937b47a68631d0a"
server_key_iv: "ab6420c9023fd4759417f93332d4453b"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.588259 15898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.595748 15898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.598037 15898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.598196 15898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.602655 15898 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:34961
I20260928 12:26:34.602715 15965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:34961 every 8 connection(s)
I20260928 12:26:34.603070 15898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:34.604540 15966 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:26:34.605233 15966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Bootstrap starting.
I20260928 12:26:34.608177 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15898
I20260928 12:26:34.608511 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:35901
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:34961
--webserver_port=43171
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:26:34.613837 15966 log.cc:824] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:34.615535 15966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:26:34.615821 15966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Bootstrap complete.
I20260928 12:26:34.618093 15966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:34.618338 15966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a292b2fcd11419697fb054f5c79fcf6, State: Initialized, Role: FOLLOWER
I20260928 12:26:34.618551 15966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:34.618644 15966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:34.618697 15966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:34.618744 15966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:26:34.619678 15966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:34.619793 15966 leader_election.cc:302] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a292b2fcd11419697fb054f5c79fcf6; no voters:
I20260928 12:26:34.619990 15966 leader_election.cc:288] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:26:34.620029 15970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:26:34.620139 15970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [term 2 LEADER]: Becoming Leader. State: Replica: 6a292b2fcd11419697fb054f5c79fcf6, State: Running, Role: LEADER
I20260928 12:26:34.620220 15970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } }
I20260928 12:26:34.620246 15966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:34.620822 15972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a292b2fcd11419697fb054f5c79fcf6. Latest consensus state: current_term: 2 leader_uuid: "6a292b2fcd11419697fb054f5c79fcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } } }
I20260928 12:26:34.620919 15972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:34.620815 15971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6a292b2fcd11419697fb054f5c79fcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a292b2fcd11419697fb054f5c79fcf6" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 34961 } } }
I20260928 12:26:34.621065 15971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:34.621085 15979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:34.621557 15979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:34.621685 15979 catalog_manager.cc:1259] Loaded cluster ID: 0e46681bc1984388a8363263df7a2f49
I20260928 12:26:34.621711 15979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:34.622843 15979 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:34.623026 15979 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a292b2fcd11419697fb054f5c79fcf6: Loaded TSK: 0
I20260928 12:26:34.623236 15979 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:26:34.693518 15968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.693754 15968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.693794 15968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.695145 15968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.695194 15968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.695223 15968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:34.696882 15968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.65.keytab
--principal=oryx/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:35901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=43171
--tserver_master_addrs=127.7.19.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.697135 15968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.697420 15968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.703408 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.703480 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.703423 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.703943 15968 server_base.cc:1034] running on GCE node
I20260928 12:26:34.704140 15968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.704398 15968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.705565 15968 hybrid_clock.cc:648] HybridClock initialized: now 1790598394705550 us; error 30 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.708606 15968 init.cc:377] Logged in from keytab as oryx/127.7.19.65@KRBTEST.COM (short username oryx)
I20260928 12:26:34.709143 15968 webserver.cc:460] Webserver started at http://127.7.19.65:43171/ using document root <none> and password file <none>
I20260928 12:26:34.709436 15968 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.709528 15968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.711114 15968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:34.711807 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.711992 15968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:34.712101 15968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "e570142e26024ba581485daa322bcf00"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "aecb58ac6012c1b74943974add3f2978"
server_key_iv: "4c0c96885c622d2d430249650a7438ef"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.712471 15968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.731922 15968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.734262 15968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.734424 15968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.734673 15968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:34.734990 15968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:34.735024 15968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.735049 15968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:34.735093 15968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.743283 15968 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:35901
I20260928 12:26:34.743407 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:35901 every 8 connection(s)
I20260928 12:26:34.743758 15968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:34.744132 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 15968
I20260928 12:26:34.744488 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:44335
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:34961
--webserver_port=33563
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:34.754505 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.744561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:56499)
0928 12:26:34.744793 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.744855 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.745687 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.747523 (+ 1836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.747527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.748098 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.748293 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.748298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.749641 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.749642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.750471 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.750473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.751109 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.751110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.751242 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.751849 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.751856 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.754299 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":113,"threads_started":1}
W20260928 12:26:34.754632 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:34.744852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:56499 (local address 127.7.19.126:34961)
0928 12:26:34.747091 (+ 2239us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.747100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.747136 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:26:34.747256 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.747263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.747316 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.747454 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.748484 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.749536 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.750578 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.751015 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.751135 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.754470 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:56499: BlockingRecv error: recv got EOF from 127.7.19.65:56499 (error 108)
Metrics: {"server-negotiator.queue_time_us":2179,"thread_start_us":74,"threads_started":1}
W20260928 12:26:34.754750 16116 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:34961 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.19.126:34961: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:34.759505 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.755018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:51347)
0928 12:26:34.755051 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.755062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.755120 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.755246 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.755247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.755377 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.755419 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.755420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.756056 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.756057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.756557 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.756558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.756974 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.756975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.757042 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.757463 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.757467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.759418 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 12:26:34.759918 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:34.755083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51347 (local address 127.7.19.126:34961)
0928 12:26:34.755144 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.755146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.755156 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:34.755177 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.755178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.755189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.755236 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.755503 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.755967 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.756638 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.756891 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.756973 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.759831 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:51347: BlockingRecv error: recv got EOF from 127.7.19.65:51347 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:34.764772 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.759828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:51101)
0928 12:26:34.759990 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.760002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.760054 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.760174 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.760175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.760342 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.760405 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.760407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.761142 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.761143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.762036 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.762037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.762372 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.762373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.762434 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.762868 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.762874 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.764659 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
W20260928 12:26:34.764825 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:34.760011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51101 (local address 127.7.19.126:34961)
0928 12:26:34.760062 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.760064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.760076 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:26:34.760102 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.760104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.760114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.760176 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.760474 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.761064 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.762114 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.762282 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.762368 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.764766 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:51101: BlockingRecv error: recv got EOF from 127.7.19.65:51101 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 12:26:34.765096 16116 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:26:34.830829 16120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.831049 16120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.831072 16120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.832173 16120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.832211 16120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.832237 16120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:34.833621 16120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.66.keytab
--principal=oryx/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:44335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=33563
--tserver_master_addrs=127.7.19.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.833797 16120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.833971 16120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.838884 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.838897 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.838901 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.839146 16120 server_base.cc:1034] running on GCE node
I20260928 12:26:34.839424 16120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.839668 16120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.840829 16120 hybrid_clock.cc:648] HybridClock initialized: now 1790598394840808 us; error 67 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.843219 16120 init.cc:377] Logged in from keytab as oryx/127.7.19.66@KRBTEST.COM (short username oryx)
I20260928 12:26:34.843662 16120 webserver.cc:460] Webserver started at http://127.7.19.66:33563/ using document root <none> and password file <none>
I20260928 12:26:34.843847 16120 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.843904 16120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.845289 16120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:34.845798 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.845988 16120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.846077 16120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "7ffbcf5cbf884f65b1eb25d7731c79a4"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "bd34335934eeb3faa6fbe3dbe484901b"
server_key_iv: "da667383d0ea4632fa4740668d293760"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.846414 16120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.861373 16120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.863722 16120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.863860 16120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.864120 16120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:34.864449 16120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:34.864506 16120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.864547 16120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:34.864574 16120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.870329 16120 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:44335
I20260928 12:26:34.870365 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:44335 every 8 connection(s)
I20260928 12:26:34.870748 16120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:26:34.877889 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:34.871472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:48093 (local address 127.7.19.126:34961)
0928 12:26:34.871543 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.871546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.871755 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:26:34.872211 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.872212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.872225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.872277 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.873318 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.873799 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.874747 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.874922 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.874973 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.877834 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:48093: BlockingRecv error: recv got EOF from 127.7.19.66:48093 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:26:34.877962 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.871452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:48093)
0928 12:26:34.871639 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.871673 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.872131 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.872375 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.872379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.873038 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.873228 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.873232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.873875 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.873876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.874672 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.874674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.875022 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.875023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.875129 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.875659 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.875667 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.877761 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
W20260928 12:26:34.878178 16250 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:34961 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.19.126:34961: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
I20260928 12:26:34.880865 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16120
I20260928 12:26:34.881166 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:44249
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:34961
--webserver_port=46383
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:34.882907 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.878409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:38567)
0928 12:26:34.878460 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.878471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.878516 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.878646 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.878647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.878769 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.878812 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.878813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.879426 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.879427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.879926 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.879927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.880287 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.880288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.880352 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.880829 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.880834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.882831 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 12:26:34.882972 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:34.878477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:38567 (local address 127.7.19.126:34961)
0928 12:26:34.878513 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.878515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.878523 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:34.878578 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.878579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.878587 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.878642 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.878885 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.879329 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.880021 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.880204 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.880286 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.882910 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:38567: BlockingRecv error: recv got EOF from 127.7.19.66:38567 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:34.888417 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:34.883358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:43419)
0928 12:26:34.883554 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:34.883566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:34.883620 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:34.883760 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:34.883761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:34.883921 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:34.883954 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.883956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.884535 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.884536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.885461 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:34.885464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.885972 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.885973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:34.886038 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.886418 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:34.886424 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:34.888325 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
W20260928 12:26:34.888562 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:34.883506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:43419 (local address 127.7.19.126:34961)
0928 12:26:34.883557 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:34.883559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:34.883604 (+ 45us) server_negotiation.cc:407] Connection header received
0928 12:26:34.883677 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:34.883678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:34.883687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:34.883758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:34.884029 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.884456 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.885712 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:34.885890 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:34.885953 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:34.888456 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:43419: BlockingRecv error: recv got EOF from 127.7.19.66:43419 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 12:26:34.888751 16250 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:26:34.963418 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:34.963637 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:34.963671 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:34.964834 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:34.964903 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:34.964947 16253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:34.966346 16253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/krb5kdc/oryx_127.7.19.67.keytab
--principal=oryx/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:44249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=46383
--tserver_master_addrs=127.7.19.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:34.966610 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:34.966835 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:34.971736 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:34.971738 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.971987 16253 server_base.cc:1034] running on GCE node
W20260928 12:26:34.971766 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:34.972184 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:34.972448 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:34.973621 16253 hybrid_clock.cc:648] HybridClock initialized: now 1790598394973595 us; error 44 us; skew 500 ppm
Sep 28 12:26:34 dist-test-slave-cb7x krb5kdc[15385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598394, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:34.975967 16253 init.cc:377] Logged in from keytab as oryx/127.7.19.67@KRBTEST.COM (short username oryx)
I20260928 12:26:34.976454 16253 webserver.cc:460] Webserver started at http://127.7.19.67:46383/ using document root <none> and password file <none>
I20260928 12:26:34.976743 16253 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:34.976825 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:34.978127 16253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:34.978729 16268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.978909 16253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:34.979000 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "357ace810fcf41c19d3f9b1fdfb54813"
format_stamp: "Formatted at 2026-09-28 12:26:34 on dist-test-slave-cb7x"
server_key: "3a6edde4f266189459b4774cc6e389be"
server_key_iv: "81b37a4fff2e4162a461e3dab3b3c00a"
server_key_version: "encryptionkey@0"
I20260928 12:26:34.979331 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:34.995978 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:34.998106 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:34.998293 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:34.998540 16253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:34.998867 16253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:34.998929 16253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:34.998974 16253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:34.999006 16253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:35.005479 16253 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:44249
I20260928 12:26:35.005517 16381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:44249 every 8 connection(s)
I20260928 12:26:35.005964 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:35.006806 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16253
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 12:26:35.014660 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:35.006728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:37767 (local address 127.7.19.126:34961)
0928 12:26:35.006799 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:35.006801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:35.007078 (+ 277us) server_negotiation.cc:407] Connection header received
0928 12:26:35.007697 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:35.007698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:35.007708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:35.007773 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:35.009004 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.009603 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.011030 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.011292 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.011342 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.014590 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:37767: BlockingRecv error: recv got EOF from 127.7.19.67:37767 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":39}
I20260928 12:26:35.014678 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:35.006720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:37767)
0928 12:26:35.006930 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:35.006981 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:35.007601 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:35.007848 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:35.007852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:35.008646 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:35.008920 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.008925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.009677 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.009678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.010954 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.010956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.011373 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.011374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.011520 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.011985 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:35.011996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:35.014504 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
W20260928 12:26:35.014896 16382 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:34961 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.19.126:34961: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
I20260928 12:26:35.015780 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:35.007350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.7.19.126:34961)
0928 12:26:35.007507 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:35.007509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:35.007607 (+ 98us) server_negotiation.cc:407] Connection header received
0928 12:26:35.007697 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:35.007697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:35.007708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:35.007774 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:35.008606 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.009339 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.010378 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.010573 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.010624 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.012125 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:35.012129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:35.012129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:35.012138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:35.014047 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:35.014441 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:35.014444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:35.014444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:35.014568 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:35.014801 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:35.014803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:35.014803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:35.015126 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:35.015274 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:35.015536 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 12:26:35.015691 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:35.019310 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:35.015153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:48415)
0928 12:26:35.015214 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:35.015239 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:35.015277 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:35.015416 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:35.015416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:35.015541 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:35.015578 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.015579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.016391 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.016392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.016973 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.016976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.017348 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.017349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.017414 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.017724 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:35.017727 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:35.019237 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 12:26:35.019357 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:35.015202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:48415 (local address 127.7.19.126:34961)
0928 12:26:35.015254 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:35.015256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:35.015265 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:35.015346 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:35.015347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:35.015355 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:35.015412 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:35.015658 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.016321 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.017066 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.017268 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.017347 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.019299 (+ 1952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:48415: BlockingRecv error: recv got EOF from 127.7.19.67:48415 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:35.023830 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:35.019562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:54853)
0928 12:26:35.019643 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:35.019653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:35.019703 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:35.019790 (+ 87us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:35.019791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:35.019891 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:35.019930 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.019931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.020783 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.020784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.021242 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.021244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.021714 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.021715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.021773 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.022172 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:35.022175 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:35.023755 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 12:26:35.023872 16119 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:35.019614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:54853 (local address 127.7.19.126:34961)
0928 12:26:35.019657 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:35.019658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:35.019691 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:26:35.019739 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:35.019740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:35.019747 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:35.019788 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:35.019986 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.020653 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.021371 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.021648 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.021707 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.023815 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:54853: BlockingRecv error: recv got EOF from 127.7.19.67:54853 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:26:35.023984 16382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:35.770363 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:35.765479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:50751)
0928 12:26:35.765707 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:35.765719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:35.765783 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:35.765986 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:35.765987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:35.766143 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:35.766190 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.766191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.766871 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.766872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.767385 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.767387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.767870 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.767871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.767940 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.768420 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:35.768425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:35.770292 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":175,"threads_started":1}
W20260928 12:26:35.770512 16388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:35.765563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:50751 (local address 127.7.19.126:34961)
0928 12:26:35.765696 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:35.765698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:35.765850 (+ 152us) server_negotiation.cc:407] Connection header received
0928 12:26:35.765884 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:35.765885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:35.765897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:35.765970 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:35.766279 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.766791 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.767524 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.767739 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.767806 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.770415 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:50751: BlockingRecv error: recv got EOF from 127.7.19.65:50751 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:35 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:35.894356 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:35.889253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:51675)
0928 12:26:35.889375 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:35.889389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:35.889460 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:35.889599 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:35.889600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:35.889766 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:35.889826 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.889828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.890543 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.890545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.891224 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:35.891227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.891598 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.891599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:35.891644 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.892208 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:35.892215 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:35.894280 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260928 12:26:35.894520 16388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:35.889324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:51675 (local address 127.7.19.126:34961)
0928 12:26:35.889413 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:35.889415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:35.889498 (+ 83us) server_negotiation.cc:407] Connection header received
0928 12:26:35.889523 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:35.889523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:35.889534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:35.889576 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:35.889913 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.890436 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.891307 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:35.891528 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:35.891608 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:35.894412 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:51675: BlockingRecv error: recv got EOF from 127.7.19.66:51675 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 12:26:36 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:36 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:36.029150 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:36.024404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:37153)
0928 12:26:36.024543 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:36.024555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:36.024629 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:36.024821 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:36.024822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:36.024985 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:36.025035 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:36.025037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.025712 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.025713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:36.026252 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:36.026254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.026626 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.026626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:36.026674 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:36.027201 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:36.027206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:36.029081 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260928 12:26:36.029222 16388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:36.024432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:37153 (local address 127.7.19.126:34961)
0928 12:26:36.024525 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:36.024527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:36.024631 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:26:36.024713 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:36.024714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:36.024727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:36.024799 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:36.025125 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.025628 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.026371 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.026558 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.026616 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:36.029173 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:37153: BlockingRecv error: recv got EOF from 127.7.19.67:37153 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 12:26:36 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:36 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:36.775521 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:36.770819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:51657)
0928 12:26:36.771002 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:36.771014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:36.771069 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:36.771199 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:36.771199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:36.771383 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:36.771445 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:36.771447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.772107 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.772108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:36.772744 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:36.772748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.773086 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.773086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:36.773137 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:36.773605 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:36.773609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:36.775425 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
W20260928 12:26:36.775573 16392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:36.770940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:51657 (local address 127.7.19.126:34961)
0928 12:26:36.771037 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:36.771039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:36.771051 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:26:36.771116 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:36.771116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:36.771126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:36.771182 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:36.771520 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.772031 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.772821 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.773018 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.773073 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:36.775501 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:51657: BlockingRecv error: recv got EOF from 127.7.19.65:51657 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
Sep 28 12:26:36 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:36 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:36.899636 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:36.894915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:44631)
0928 12:26:36.895063 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:36.895073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:36.895132 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:36.895260 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:36.895261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:36.895423 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:36.895471 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:36.895473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.896133 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.896134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:36.896727 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:36.896729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.897102 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.897103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:36.897153 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:36.897609 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:36.897616 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:36.899517 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260928 12:26:36.899825 16392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:36.894986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:44631 (local address 127.7.19.126:34961)
0928 12:26:36.895070 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:36.895071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:36.895135 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:26:36.895177 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:36.895177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:36.895188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:36.895241 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:36.895545 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.896050 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.896814 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:36.897011 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:36.897067 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:36.899763 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:44631: BlockingRecv error: recv got EOF from 127.7.19.66:44631 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 12:26:37 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:37 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:37.034423 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:37.029739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:34867)
0928 12:26:37.029873 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:37.029908 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:37.029987 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:37.030180 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:37.030181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:37.030361 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:37.030406 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:37.030408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.031102 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.031103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:37.031624 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:37.031626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.031998 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.031999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:37.032063 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:37.032574 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:37.032578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:37.034318 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260928 12:26:37.034510 16392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:37.029855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:34867 (local address 127.7.19.126:34961)
0928 12:26:37.029937 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:37.029939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:37.029967 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:26:37.030076 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:37.030077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:37.030101 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:37.030171 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:37.030514 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.031007 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.031726 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.031908 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.031996 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:37.034401 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:34867: BlockingRecv error: recv got EOF from 127.7.19.67:34867 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 12:26:37 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:37 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:37.781067 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:37.776088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:35953)
0928 12:26:37.776250 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:37.776263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:37.776324 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:37.776465 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:37.776466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:37.776645 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:37.776769 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:37.776772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.777689 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.777690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:37.778233 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:37.778234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.778617 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.778618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:37.778660 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:37.779140 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:37.779144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:37.780968 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
W20260928 12:26:37.781121 16396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:37.776176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:35953 (local address 127.7.19.126:34961)
0928 12:26:37.776313 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:37.776315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:37.776323 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:37.776384 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:37.776385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:37.776396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:37.776445 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:37.776907 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.777613 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.778336 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.778543 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.778604 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:37.781049 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:35953: BlockingRecv error: recv got EOF from 127.7.19.65:35953 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
Sep 28 12:26:37 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:37 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:37.904661 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:37.900203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:40179)
0928 12:26:37.900347 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:37.900357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:37.900424 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:37.900559 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:37.900560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:37.900790 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:37.900845 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:37.900847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.901491 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.901492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:37.902045 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:37.902047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.902359 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.902360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:37.902401 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:37.902857 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:37.902861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:37.904590 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260928 12:26:37.904948 16396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:37.900255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:40179 (local address 127.7.19.126:34961)
0928 12:26:37.900337 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:37.900339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:37.900423 (+ 84us) server_negotiation.cc:407] Connection header received
0928 12:26:37.900469 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:37.900470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:37.900480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:37.900534 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:37.900912 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.901420 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.902114 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:37.902291 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:37.902341 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:37.904736 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:40179: BlockingRecv error: recv got EOF from 127.7.19.66:40179 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 12:26:38 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:38 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:38.040241 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:38.034908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:50929)
0928 12:26:38.035083 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:38.035099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:38.035200 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:38.035419 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:38.035421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:38.035619 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:38.035664 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:38.035666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.036405 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.036406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:38.037032 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:38.037034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.037445 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.037446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:38.037513 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:38.038043 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:38.038048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:38.040148 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
W20260928 12:26:38.040269 16396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:38.035002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:50929 (local address 127.7.19.126:34961)
0928 12:26:38.035088 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:38.035091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:38.035180 (+ 89us) server_negotiation.cc:407] Connection header received
0928 12:26:38.035321 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:38.035322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:38.035338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:38.035391 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:38.035757 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.036298 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.037127 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.037349 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.037444 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:38.040221 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:50929: BlockingRecv error: recv got EOF from 127.7.19.67:50929 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 12:26:38 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:38 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:38.786099 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:38.781502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:34051)
0928 12:26:38.781626 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:38.781638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:38.781694 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:38.781878 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:38.781879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:38.782056 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:38.782106 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:38.782108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.782818 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.782819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:38.783364 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:38.783366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.783723 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.783725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:38.783771 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:38.784244 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:38.784249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:38.786030 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
W20260928 12:26:38.786238 16400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:38.781606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:34051 (local address 127.7.19.126:34961)
0928 12:26:38.781733 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:38.781735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:38.781751 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:26:38.781772 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:38.781773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:38.781782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:38.781866 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:38.782173 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.782740 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.783447 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.783648 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.783706 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:38.786135 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:34051: BlockingRecv error: recv got EOF from 127.7.19.65:34051 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
Sep 28 12:26:38 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:38 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:38.909866 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:38.905276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:56143)
0928 12:26:38.905433 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:38.905444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:38.905491 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:38.905635 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:38.905636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:38.905793 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:38.905851 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:38.905852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.906511 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.906512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:38.907101 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:38.907104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.907439 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.907440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:38.907491 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:38.907944 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:38.907949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:38.909787 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260928 12:26:38.909974 16400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:38.905414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:56143 (local address 127.7.19.126:34961)
0928 12:26:38.905473 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:38.905480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:38.905489 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:38.905550 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:38.905551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:38.905561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:38.905616 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:38.905925 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.906409 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.907176 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:38.907354 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:38.907407 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:38.909879 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:56143: BlockingRecv error: recv got EOF from 127.7.19.66:56143 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 12:26:39 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:39 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:39.045980 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:39.040903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:41377)
0928 12:26:39.041036 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:39.041048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:39.041114 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:39.041285 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:39.041286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:39.041448 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:39.041496 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:39.041498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.042537 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.042538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:39.043065 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:39.043067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.043419 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.043420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:39.043460 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:39.043985 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:39.043989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:39.045922 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260928 12:26:39.046108 16400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:39.040976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:41377 (local address 127.7.19.126:34961)
0928 12:26:39.041047 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:39.041050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:39.041119 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:26:39.041162 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:39.041163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:39.041176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:39.041240 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:39.041577 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.042472 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.043145 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.043354 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.043411 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:39.046025 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:41377: BlockingRecv error: recv got EOF from 127.7.19.67:41377 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:26:39 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:39 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:39.800911 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:39.786629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:50053)
0928 12:26:39.786828 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:39.786840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:39.786914 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:39.787096 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:39.787097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:39.787300 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:39.787361 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:39.787364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.788120 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.788121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:39.788768 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:39.788771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.789171 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.789172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:39.789244 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:39.798868 (+ 9624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:39.798874 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:39.800837 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":131,"threads_started":1}
W20260928 12:26:39.801046 16404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:39.786681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:50053 (local address 127.7.19.126:34961)
0928 12:26:39.786820 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:39.786822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:39.786894 (+ 72us) server_negotiation.cc:407] Connection header received
0928 12:26:39.786973 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:39.786974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:39.786991 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:39.787076 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:39.787457 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.788025 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.788866 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.789086 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.789172 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:39.800959 (+ 11787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:50053: BlockingRecv error: recv got EOF from 127.7.19.65:50053 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
Sep 28 12:26:39 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:39 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:39.915009 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:39.910295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:49627)
0928 12:26:39.910452 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:39.910464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:39.910527 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:39.910704 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:39.910706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:39.910941 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:39.910989 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:39.910991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.911666 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.911667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:39.912203 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:39.912204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.912566 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.912566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:39.912610 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:39.913138 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:39.913143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:39.914950 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
W20260928 12:26:39.915117 16404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:39.910427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49627 (local address 127.7.19.126:34961)
0928 12:26:39.910473 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:39.910475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:39.910508 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:26:39.910600 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:39.910601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:39.910616 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:39.910682 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:39.911055 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.911597 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.912284 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:39.912492 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:39.912555 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:39.915041 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:49627: BlockingRecv error: recv got EOF from 127.7.19.66:49627 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 12:26:40 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:40 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:40.051651 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:40.046508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:37931)
0928 12:26:40.046802 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:40.046815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:40.046906 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:40.047054 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:40.047055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:40.047238 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:40.047282 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:40.047283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.047978 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.047979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:40.048523 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:40.048525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.048979 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.048980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:40.049059 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:40.049544 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:40.049549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:40.051552 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":109,"threads_started":1}
W20260928 12:26:40.051704 16404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:40.046640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:37931 (local address 127.7.19.126:34961)
0928 12:26:40.046805 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:40.046808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:40.046887 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:26:40.046983 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:40.046985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:40.046998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:40.047050 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:40.047361 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.047885 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.048608 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.048886 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.048964 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:40.051640 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:37931: BlockingRecv error: recv got EOF from 127.7.19.67:37931 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 28 12:26:40 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:40 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:40.806285 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:40.801446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:43459)
0928 12:26:40.801591 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:40.801609 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:40.801658 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:40.801836 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:40.801837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:40.802013 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:40.802062 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:40.802063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.802759 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.802760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:40.803330 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:40.803331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.803746 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.803747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:40.803795 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:40.804309 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:40.804313 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:40.806216 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260928 12:26:40.806454 16408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:40.801553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:43459 (local address 127.7.19.126:34961)
0928 12:26:40.801671 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:40.801678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:40.801687 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:40.801710 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:40.801710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:40.801721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:40.801791 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:40.802155 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.802675 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.803450 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.803665 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.803730 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:40.806341 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:43459: BlockingRecv error: recv got EOF from 127.7.19.65:43459 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
Sep 28 12:26:40 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:40 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:40.920543 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:40.915477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:47467)
0928 12:26:40.915630 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:40.915643 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:40.915710 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:40.915894 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:40.915894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:40.916073 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:40.916118 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:40.916120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.916983 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.916984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:40.917531 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:40.917533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.917925 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.917926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:40.917983 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:40.918415 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:40.918420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:40.920484 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
W20260928 12:26:40.920740 16408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:40.915545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:47467 (local address 127.7.19.126:34961)
0928 12:26:40.915636 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:40.915638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:40.915689 (+ 51us) server_negotiation.cc:407] Connection header received
0928 12:26:40.915823 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:40.915824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:40.915837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:40.915889 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:40.916220 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.916894 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.917620 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:40.917835 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:40.917925 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:40.920596 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:47467: BlockingRecv error: recv got EOF from 127.7.19.66:47467 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 12:26:41 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:41 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:41.056867 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:41.052208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:33283)
0928 12:26:41.052379 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:41.052389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:41.052438 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:41.052580 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:41.052581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:41.052808 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:41.052857 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:41.052859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.053488 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.053488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:41.054007 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:41.054008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.054311 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.054312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:41.054355 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:41.054781 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:41.054785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:41.056799 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
W20260928 12:26:41.056957 16408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:41.052317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:33283 (local address 127.7.19.126:34961)
0928 12:26:41.052408 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:41.052410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:41.052435 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:26:41.052503 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:41.052504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:41.052516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:41.052558 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:41.052929 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.053418 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.054075 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.054249 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.054298 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:41.056890 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:33283: BlockingRecv error: recv got EOF from 127.7.19.67:33283 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 12:26:41 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:41 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:41.811913 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:41.806848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:57711)
0928 12:26:41.807022 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:41.807033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:41.807092 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:41.807217 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:41.807218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:41.807437 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:41.807489 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:41.807491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.808146 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.808146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:41.808782 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:41.808784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.809164 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.809165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:41.809255 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:41.809873 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:41.809878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:41.811848 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
W20260928 12:26:41.812045 16412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:41.806912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:57711 (local address 127.7.19.126:34961)
0928 12:26:41.807041 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:41.807042 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:41.807077 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:26:41.807137 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:41.807138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:41.807149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:41.807205 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:41.807560 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.808068 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.808868 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.809068 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.809157 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:41.811959 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:57711: BlockingRecv error: recv got EOF from 127.7.19.65:57711 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
Sep 28 12:26:41 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:41 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:41.925767 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:41.921025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:33777)
0928 12:26:41.921192 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:41.921206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:41.921315 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:41.921459 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:41.921460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:41.921664 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:41.921713 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:41.921715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.922412 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.922413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:41.922939 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:41.922940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.923254 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.923255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:41.923294 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:41.923826 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:41.923830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:41.925709 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
W20260928 12:26:41.925904 16412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:41.921097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:33777 (local address 127.7.19.126:34961)
0928 12:26:41.921177 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:41.921179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:41.921305 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:26:41.921385 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:41.921386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:41.921397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:41.921439 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:41.921802 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.922319 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.923011 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:41.923192 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:41.923244 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:41.925817 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:33777: BlockingRecv error: recv got EOF from 127.7.19.66:33777 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 12:26:42 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:42 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:42.062304 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:42.057411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:55177)
0928 12:26:42.057561 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:42.057571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:42.057629 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:42.057754 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:42.057755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:42.057920 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:42.057965 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:42.057967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.058960 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.058961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:42.059484 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:42.059485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.059800 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.059800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:42.059839 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:42.060283 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:42.060287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:42.062245 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
W20260928 12:26:42.062446 16412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:42.057505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:55177 (local address 127.7.19.126:34961)
0928 12:26:42.057602 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:42.057604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:42.057616 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:26:42.057685 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:42.057686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:42.057698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:42.057752 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:42.058093 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.058870 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.059558 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.059737 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.059799 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:42.062353 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:55177: BlockingRecv error: recv got EOF from 127.7.19.67:55177 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 12:26:42 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:42 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:42.817420 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:42.812469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:45205)
0928 12:26:42.812611 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:42.812630 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:42.812711 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:42.812943 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:42.812944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:42.813108 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:42.813158 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:42.813160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.813823 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.813824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:42.814353 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:42.814354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.814695 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.814695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:42.814736 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:42.815395 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:42.815401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:42.817321 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
W20260928 12:26:42.817466 16416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:42.812655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:45205 (local address 127.7.19.126:34961)
0928 12:26:42.812819 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:42.812821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:42.812830 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:42.812852 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:42.812853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:42.812863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:42.812924 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:42.813230 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.813752 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.814430 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.814628 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.814685 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:42.817397 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:45205: BlockingRecv error: recv got EOF from 127.7.19.65:45205 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
Sep 28 12:26:42 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:42 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:42.931322 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:42.926278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:37061)
0928 12:26:42.926586 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:42.926598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:42.926686 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:42.926851 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:42.926852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:42.927037 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:42.927083 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:42.927085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.927908 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.927909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:42.928465 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:42.928467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.928937 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.928938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:42.928998 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:42.929474 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:42.929478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:42.931261 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":188,"threads_started":1}
W20260928 12:26:42.931463 16416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:42.926472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:37061 (local address 127.7.19.126:34961)
0928 12:26:42.926572 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:42.926575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:42.926679 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:26:42.926747 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:42.926749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:42.926763 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:42.926838 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:42.927167 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.927821 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.928552 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:42.928842 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:42.928938 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:42.931381 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:37061: BlockingRecv error: recv got EOF from 127.7.19.66:37061 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 12:26:43 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:43 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:43.067634 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:43.062798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:42727)
0928 12:26:43.063087 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:43.063098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:43.063191 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:43.063417 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:43.063418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:43.063599 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:43.063646 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:43.063648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.064426 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.064427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:43.064982 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:43.064984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.065352 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.065353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:43.065416 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:43.065841 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:43.065845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:43.067540 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":96,"threads_started":1}
W20260928 12:26:43.067682 16416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:43.062908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:42727 (local address 127.7.19.126:34961)
0928 12:26:43.063083 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:43.063085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:43.063172 (+ 87us) server_negotiation.cc:407] Connection header received
0928 12:26:43.063326 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:43.063327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:43.063341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:43.063407 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:43.063727 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.064334 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.065077 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.065263 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.065351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:43.067616 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:42727: BlockingRecv error: recv got EOF from 127.7.19.67:42727 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 28 12:26:43 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:43 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:43.823148 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:43.817895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:53445)
0928 12:26:43.818062 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:43.818073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:43.818122 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:43.818263 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:43.818264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:43.818427 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:43.818475 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:43.818477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.819151 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.819152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:43.820055 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:43.820058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.820461 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.820462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:43.820510 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:43.821077 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:43.821082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:43.823089 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
W20260928 12:26:43.823312 16420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:43.818004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:53445 (local address 127.7.19.126:34961)
0928 12:26:43.818127 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:43.818128 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:43.818138 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:43.818163 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:43.818163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:43.818173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:43.818240 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:43.818541 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.819073 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.820121 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.820389 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.820444 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:43.823226 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:53445: BlockingRecv error: recv got EOF from 127.7.19.65:53445 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
Sep 28 12:26:43 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:43 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:43.936291 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:43.931817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:36001)
0928 12:26:43.931977 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:43.931997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:43.932057 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:43.932192 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:43.932192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:43.932359 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:43.932406 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:43.932408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.933078 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.933079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:43.933644 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:43.933647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.933972 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.933973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:43.934011 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:43.934460 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:43.934465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:43.936232 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
W20260928 12:26:43.936419 16420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:43.931943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:36001 (local address 127.7.19.126:34961)
0928 12:26:43.932009 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:43.932011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:43.932043 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:26:43.932115 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:43.932116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:43.932126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:43.932174 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:43.932475 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.933003 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.933723 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:43.933908 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:43.933958 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:43.936322 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:36001: BlockingRecv error: recv got EOF from 127.7.19.66:36001 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 12:26:44 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:44 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:44.072954 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:44.068137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:46769)
0928 12:26:44.068317 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:44.068329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:44.068395 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:44.068526 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:44.068527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:44.068728 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:44.068773 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:44.068775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.069518 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.069519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:44.070045 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:44.070047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.070418 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.070419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:44.070483 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:44.070953 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:44.070958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:44.072885 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260928 12:26:44.073050 16420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:44.068234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:46769 (local address 127.7.19.126:34961)
0928 12:26:44.068306 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:44.068308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:44.068387 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:26:44.068457 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:44.068458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:44.068470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:44.068510 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:44.068854 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.069364 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.070125 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.070327 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.070416 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:44.072982 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:46769: BlockingRecv error: recv got EOF from 127.7.19.67:46769 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 12:26:44 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:44 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:44.833909 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:44.823564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:52873)
0928 12:26:44.823679 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:44.823691 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:44.823763 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:44.823893 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:44.823894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:44.824061 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:44.824111 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:44.824113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.824848 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.824853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:44.825485 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:44.825488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.825866 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.825866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:44.825906 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:44.832089 (+ 6183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:44.832093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:44.833850 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
W20260928 12:26:44.834079 16424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:44.823634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:52873 (local address 127.7.19.126:34961)
0928 12:26:44.823767 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:44.823769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:44.823780 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:26:44.823804 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:44.823805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:44.823816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:44.823874 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:44.824183 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.824774 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.825583 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.825798 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.825856 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:44.833978 (+ 8122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:52873: BlockingRecv error: recv got EOF from 127.7.19.65:52873 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
Sep 28 12:26:44 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:44 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:44.941016 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:44.936824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:38119)
0928 12:26:44.936950 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:44.936960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:44.937019 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:44.937125 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:44.937126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:44.937295 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:44.937343 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:44.937345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.937949 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.937950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:44.938483 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:44.938484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.938786 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.938786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:44.938823 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:44.939260 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:44.939264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:44.940959 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260928 12:26:44.941238 16424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:44.936876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:38119 (local address 127.7.19.126:34961)
0928 12:26:44.936930 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:44.936932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:44.937020 (+ 88us) server_negotiation.cc:407] Connection header received
0928 12:26:44.937064 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:44.937064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:44.937075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:44.937122 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:44.937402 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.937879 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.938553 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:44.938727 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:44.938775 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:44.941058 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:38119: BlockingRecv error: recv got EOF from 127.7.19.66:38119 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 12:26:45 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:45 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:45.078274 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:45.073459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:50403)
0928 12:26:45.073675 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:45.073689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:45.073753 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:45.073888 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:45.073889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:45.074062 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:45.074112 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:45.074114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.074736 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.074737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:45.075329 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:45.075331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.075663 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.075664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:45.075708 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:45.076166 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:45.076171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:45.078176 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20260928 12:26:45.078328 16424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:45.073658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:50403 (local address 127.7.19.126:34961)
0928 12:26:45.073720 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:45.073725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:45.073749 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:26:45.073813 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:45.073813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:45.073825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:45.073867 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:45.074170 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.074656 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.075407 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.075590 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.075644 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:45.078262 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:50403: BlockingRecv error: recv got EOF from 127.7.19.67:50403 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 12:26:45 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:45 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:45.838822 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:45.834411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:42333)
0928 12:26:45.834560 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:45.834579 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:45.834627 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:45.834774 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:45.834775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:45.834937 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:45.834982 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:45.834983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.835674 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.835675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:45.836207 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:45.836209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.836525 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.836525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:45.836562 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:45.837045 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:45.837050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:45.838755 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260928 12:26:45.838950 16428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:45.834509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:42333 (local address 127.7.19.126:34961)
0928 12:26:45.834633 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:45.834635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:45.834645 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:45.834668 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:45.834668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:45.834678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:45.834741 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:45.835056 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.835601 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.836277 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.836464 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.836513 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:45.838843 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:42333: BlockingRecv error: recv got EOF from 127.7.19.65:42333 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
Sep 28 12:26:45 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:45 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:45.946291 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:45.941640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:51079)
0928 12:26:45.941822 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:45.941833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:45.941914 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:45.942060 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:45.942061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:45.942249 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:45.942293 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:45.942294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.942970 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.942972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:45.943525 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:45.943527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.943906 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.943907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:45.943963 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:45.944396 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:45.944400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:45.946221 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
W20260928 12:26:45.946415 16428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:45.941700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:51079 (local address 127.7.19.126:34961)
0928 12:26:45.941778 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:45.941780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:45.941898 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:26:45.941985 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:45.941986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:45.941998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:45.942044 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:45.942375 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.942882 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.943622 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:45.943835 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:45.943908 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:45.946316 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:51079: BlockingRecv error: recv got EOF from 127.7.19.66:51079 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 12:26:46 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:46 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:46.083758 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:46.078855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:58091)
0928 12:26:46.079017 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:46.079027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:46.079083 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:46.079217 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:46.079217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:46.079394 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:46.079449 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:46.079451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.080094 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.080095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:46.080619 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:46.080620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.081013 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.081014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:46.081057 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:46.081557 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:46.081561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:46.083690 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
W20260928 12:26:46.083827 16428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:46.078920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:58091 (local address 127.7.19.126:34961)
0928 12:26:46.079011 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:46.079013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:46.079070 (+ 57us) server_negotiation.cc:407] Connection header received
0928 12:26:46.079140 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:46.079141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:46.079153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:46.079196 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:46.079513 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.080000 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.080768 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.080943 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.081000 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:46.083778 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:58091: BlockingRecv error: recv got EOF from 127.7.19.67:58091 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 12:26:46 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:46 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:46.843474 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:46.839227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:57679)
0928 12:26:46.839358 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:46.839371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:46.839440 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:46.839568 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:46.839568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:46.839750 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:46.839797 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:46.839798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.840426 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.840427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:46.840977 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:46.840979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.841306 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.841307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:46.841347 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:46.841774 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:46.841778 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:46.843415 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
W20260928 12:26:46.843575 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:46.839307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:57679 (local address 127.7.19.126:34961)
0928 12:26:46.839464 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:46.839466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:46.839474 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:46.839496 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:46.839496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:46.839507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:46.839564 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:46.839859 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.840348 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.841048 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.841241 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.841295 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:46.843502 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:57679: BlockingRecv error: recv got EOF from 127.7.19.65:57679 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
Sep 28 12:26:46 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:46 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:46.951495 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:46.946739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:55633)
0928 12:26:46.946899 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:46.946910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:46.946978 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:46.947223 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:46.947224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:46.947497 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:46.947554 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:46.947557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.948318 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.948318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:46.948931 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:46.948933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.949276 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.949277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:46.949316 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:46.949739 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:46.949743 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:46.951419 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
W20260928 12:26:46.951557 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:46.946854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:55633 (local address 127.7.19.126:34961)
0928 12:26:46.946908 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:46.946910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:46.946964 (+ 54us) server_negotiation.cc:407] Connection header received
0928 12:26:46.947065 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:46.947065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:46.947076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:46.947152 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:46.947740 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.948249 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.949015 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:46.949211 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:46.949264 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:46.951512 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:55633: BlockingRecv error: recv got EOF from 127.7.19.66:55633 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 12:26:47 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:47 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:47.088989 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:47.084295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:60231)
0928 12:26:47.084493 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:47.084503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:47.084559 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:47.084700 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:47.084700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:47.084862 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:47.084906 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:47.084907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.085513 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.085514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:47.086027 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:47.086028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.086380 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.086381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:47.086422 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:47.087011 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:47.087016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:47.088909 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":116,"threads_started":1}
W20260928 12:26:47.089085 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:47.084392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:60231 (local address 127.7.19.126:34961)
0928 12:26:47.084455 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:47.084457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:47.084561 (+ 104us) server_negotiation.cc:407] Connection header received
0928 12:26:47.084619 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:47.084620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:47.084632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:47.084686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:47.084967 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.085444 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.086098 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.086281 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.086330 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:47.089016 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:60231: BlockingRecv error: recv got EOF from 127.7.19.67:60231 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 12:26:47 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:47 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:47.848809 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:47.844023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:50091)
0928 12:26:47.844191 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:47.844201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:47.844258 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:47.844428 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:47.844429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:47.844592 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:47.844639 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:47.844641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.845385 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.845386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:47.845908 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:47.845909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.846257 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.846258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:47.846295 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:47.846942 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:47.846947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:47.848743 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
W20260928 12:26:47.848973 16436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:47.844134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:50091 (local address 127.7.19.126:34961)
0928 12:26:47.844266 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:47.844268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:47.844278 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:47.844303 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:47.844304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:47.844314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:47.844416 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:47.844738 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.845309 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.845986 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.846192 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.846247 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:47.848839 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:50091: BlockingRecv error: recv got EOF from 127.7.19.65:50091 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
Sep 28 12:26:47 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:47 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:47.956805 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:47.951964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:53491)
0928 12:26:47.952114 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:47.952124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:47.952175 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:47.952342 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:47.952343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:47.952512 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:47.952558 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:47.952559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.953287 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.953288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:47.953850 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:47.953852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.954218 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.954219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:47.954271 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:47.954882 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:47.954887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:47.956660 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260928 12:26:47.956841 16436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:47.952112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:53491 (local address 127.7.19.126:34961)
0928 12:26:47.952183 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:47.952185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:47.952201 (+ 16us) server_negotiation.cc:407] Connection header received
0928 12:26:47.952227 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:47.952228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:47.952272 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:47.952323 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:47.952652 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.953220 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.953935 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:47.954143 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:47.954237 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:47.956777 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:53491: BlockingRecv error: recv got EOF from 127.7.19.66:53491 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 12:26:48 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:48 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:48.094623 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:48.089516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:40849)
0928 12:26:48.089717 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:48.089730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:48.089839 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:48.090032 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:48.090034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:48.090229 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:48.090273 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:48.090275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.091163 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.091164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:48.091804 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:48.091807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.092165 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.092166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:48.092222 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:48.092745 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:48.092752 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:48.094528 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
W20260928 12:26:48.094672 16436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:48.089608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:40849 (local address 127.7.19.126:34961)
0928 12:26:48.089706 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:48.089709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:48.089818 (+ 109us) server_negotiation.cc:407] Connection header received
0928 12:26:48.089926 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:48.089927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:48.089943 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:48.090025 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:48.090370 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.090955 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.091894 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.092089 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.092167 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:48.094606 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:40849: BlockingRecv error: recv got EOF from 127.7.19.67:40849 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 12:26:48 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:48 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:48.853945 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:48.849234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:55133)
0928 12:26:48.849368 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:48.849378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:48.849427 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:48.849612 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:48.849613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:48.849818 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:48.849865 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:48.849867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.850626 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.850627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:48.851168 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:48.851170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.851540 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.851541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:48.851606 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:48.852056 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:48.852061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:48.853849 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
W20260928 12:26:48.854000 16440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:48.849338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:55133 (local address 127.7.19.126:34961)
0928 12:26:48.849464 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:48.849465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:48.849473 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:48.849494 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:48.849495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:48.849511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:48.849590 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:48.849952 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.850542 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.851256 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.851459 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.851541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:48.853929 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:55133: BlockingRecv error: recv got EOF from 127.7.19.65:55133 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
Sep 28 12:26:48 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:48 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:48.961751 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:48.957293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:58997)
0928 12:26:48.957427 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:48.957438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:48.957505 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:48.957664 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:48.957665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:48.957899 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:48.957983 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:48.957985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.958618 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.958619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:48.959142 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:48.959143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.959458 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.959459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:48.959501 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:48.959926 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:48.959942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:48.961693 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260928 12:26:48.961884 16440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:48.957364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:58997 (local address 127.7.19.126:34961)
0928 12:26:48.957428 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:48.957430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:48.957509 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:26:48.957562 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:48.957563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:48.957576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:48.957642 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:48.958052 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.958551 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.959210 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:48.959391 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:48.959441 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:48.961802 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:58997: BlockingRecv error: recv got EOF from 127.7.19.66:58997 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 12:26:49 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:49 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.67@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:49.099808 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:49.095036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.67:60513)
0928 12:26:49.095174 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:49.095190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:49.095238 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:49.095417 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:49.095418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:49.095590 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:49.095635 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:49.095637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.096251 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.096251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:49.096824 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:49.096826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.097305 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.097307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:49.097373 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:49.097900 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:49.097906 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:49.099712 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260928 12:26:49.099855 16440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:49.095141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:60513 (local address 127.7.19.126:34961)
0928 12:26:49.095218 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:49.095219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:49.095228 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:49.095305 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:49.095305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:49.095318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:49.095398 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:49.095702 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.096186 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.096911 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.097179 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.097297 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:49.099789 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.67:60513: BlockingRecv error: recv got EOF from 127.7.19.67:60513 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 12:26:49 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:49 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.65@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:49.866570 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:49.854455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.65:48893)
0928 12:26:49.854606 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:49.854617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:49.854672 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:49.854802 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:49.854802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:49.854963 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:49.855010 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:49.855012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.855630 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.855631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:49.856163 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:49.856165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.856506 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.856507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:49.856554 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:49.864530 (+ 7976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:49.864535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:49.866496 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260928 12:26:49.866681 16444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:49.854540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:48893 (local address 127.7.19.126:34961)
0928 12:26:49.854659 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:49.854661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:49.854678 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:26:49.854717 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:49.854718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:49.854728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:49.854785 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:49.855072 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.855562 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.856238 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.856433 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.856492 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:49.866600 (+ 10108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.65:48893: BlockingRecv error: recv got EOF from 127.7.19.65:48893 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
Sep 28 12:26:49 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 12:26:49 dist-test-slave-cb7x krb5kdc[15385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.19.66@KRBTEST.COM for oryx/127.7.19.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 12:26:49.967151 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:49.962289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:34961 (local address 127.7.19.66:45063)
0928 12:26:49.962488 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:49.962499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:49.962596 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:49.962771 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:49.962772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:49.962956 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:49.963005 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:49.963007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.963704 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.963705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:49.964266 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:49.964268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.964701 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.964702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:49.964770 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:49.965301 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:49.965306 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:49.967053 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.19.126:34961: Server oryx/127.7.19.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
W20260928 12:26:49.967196 16444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:49.962380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:45063 (local address 127.7.19.126:34961)
0928 12:26:49.962476 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:49.962478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:49.962578 (+ 100us) server_negotiation.cc:407] Connection header received
0928 12:26:49.962688 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:49.962689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:49.962701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:49.962760 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:49.963091 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.963614 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.964363 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:49.964578 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:49.964691 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:49.967132 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.19.66:45063: BlockingRecv error: recv got EOF from 127.7.19.66:45063 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 12:26:50.014206 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15968
I20260928 12:26:50.019104 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16120
I20260928 12:26:50.023631 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16253
I20260928 12:26:50.027752 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 15898
2026-09-28T12:26:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17437 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T12:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:50Z Disabled control of system clock
I20260928 12:26:50.054544 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:35597
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:35597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 12:26:50.135769 16449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:50.135986 16449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:50.136015 16449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:50.137426 16449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:50.137475 16449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:50.137487 16449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:50.137501 16449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:50.137512 16449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:50.138841 16449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:35597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:35597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:50.139038 16449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:50.139271 16449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:50.144529 16455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.144510 16457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.144512 16454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.144543 16449 server_base.cc:1034] running on GCE node
I20260928 12:26:50.144974 16449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:50.145304 16449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:50.146476 16449 hybrid_clock.cc:648] HybridClock initialized: now 1790598410146465 us; error 41 us; skew 500 ppm
I20260928 12:26:50.147701 16449 webserver.cc:460] Webserver started at http://127.7.19.126:41423/ using document root <none> and password file <none>
I20260928 12:26:50.147951 16449 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:50.148051 16449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:50.148223 16449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:50.149371 16449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "f0ff6b436b1c4781982272e17749b853"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "1dab5827982340a83060c630fb88d446"
server_key_iv: "43d7d58d551bbec57acccba4578177e9"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.149736 16449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "f0ff6b436b1c4781982272e17749b853"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "1dab5827982340a83060c630fb88d446"
server_key_iv: "43d7d58d551bbec57acccba4578177e9"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.151046 16449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:50.151790 16463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.151990 16449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:50.152091 16449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "f0ff6b436b1c4781982272e17749b853"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "1dab5827982340a83060c630fb88d446"
server_key_iv: "43d7d58d551bbec57acccba4578177e9"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.152171 16449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:50.165347 16449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:50.165637 16449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:50.165782 16449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:50.169847 16449 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:35597
I20260928 12:26:50.169886 16515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:35597 every 8 connection(s)
I20260928 12:26:50.170234 16449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:50.170890 16516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:50.173554 16516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853: Bootstrap starting.
I20260928 12:26:50.174194 16516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:50.174482 16516 log.cc:824] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:50.175235 16516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853: No bootstrap required, opened a new log
I20260928 12:26:50.177033 16516 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff6b436b1c4781982272e17749b853" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35597 } }
I20260928 12:26:50.177170 16516 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:50.177192 16516 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ff6b436b1c4781982272e17749b853, State: Initialized, Role: FOLLOWER
I20260928 12:26:50.177345 16516 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff6b436b1c4781982272e17749b853" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35597 } }
I20260928 12:26:50.177425 16516 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:50.177448 16516 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:50.177493 16516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:50.177974 16516 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff6b436b1c4781982272e17749b853" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35597 } }
I20260928 12:26:50.178062 16516 leader_election.cc:302] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0ff6b436b1c4781982272e17749b853; no voters:
I20260928 12:26:50.178179 16516 leader_election.cc:288] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:50.178258 16519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:50.178424 16519 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [term 1 LEADER]: Becoming Leader. State: Replica: f0ff6b436b1c4781982272e17749b853, State: Running, Role: LEADER
I20260928 12:26:50.178458 16516 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:50.178524 16519 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff6b436b1c4781982272e17749b853" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35597 } }
I20260928 12:26:50.178699 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16449
I20260928 12:26:50.178815 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:50.178865 16520 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0ff6b436b1c4781982272e17749b853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff6b436b1c4781982272e17749b853" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35597 } } }
I20260928 12:26:50.178963 16520 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:50.179051 7245 external_mini_cluster.cc:1428] Setting key 3781720db2096a821a4aec1ad1a2fe6c
I20260928 12:26:50.179188 16523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:50.179214 16521 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0ff6b436b1c4781982272e17749b853. Latest consensus state: current_term: 1 leader_uuid: "f0ff6b436b1c4781982272e17749b853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ff6b436b1c4781982272e17749b853" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 35597 } } }
I20260928 12:26:50.179391 16521 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:50.179730 16523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:50.181476 16523 catalog_manager.cc:1347] Generated new cluster ID: 7c00ef1376b84c51b10d1092fa7c1598
I20260928 12:26:50.181545 16523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:50.193358 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.179643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.7.19.126:35597)
0928 12:26:50.180462 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.180471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.180502 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:26:50.180608 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.180615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.180705 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.181102 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.181218 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.189880 (+ 8662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.191083 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.191701 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.191815 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.192601 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:50.192605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:50.192606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:50.192836 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:50.192877 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:50.192892 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:50.192952 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:50.193052 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:26:50.193127 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":84,"threads_started":1}
I20260928 12:26:50.198639 16523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:50.199286 16523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:50.202499 16523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0ff6b436b1c4781982272e17749b853: Generated new TSK 0
I20260928 12:26:50.202663 16523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:50.204459 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35597
--builtin_ntp_servers=127.7.19.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:26:50.286003 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:50.286232 16540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:50.286270 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:50.287395 16540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:50.287453 16540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:50.287495 16540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:50.288959 16540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:50.289215 16540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:50.289502 16540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:50.295420 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.295552 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.295428 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.295691 16540 server_base.cc:1034] running on GCE node
I20260928 12:26:50.295882 16540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:50.296088 16540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:50.297235 16540 hybrid_clock.cc:648] HybridClock initialized: now 1790598410297177 us; error 63 us; skew 500 ppm
I20260928 12:26:50.298735 16540 webserver.cc:460] Webserver started at http://127.7.19.65:45449/ using document root <none> and password file <none>
I20260928 12:26:50.298979 16540 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:50.299049 16540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:50.299194 16540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:50.300319 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "af631e8eca694aa9acf38735d0b5fb77"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "3698b2ee97d26eb53f7c6acbeb34978a"
server_key_iv: "860799fd4370d3a2fce6ddc8c89e26d1"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.300750 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "af631e8eca694aa9acf38735d0b5fb77"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "3698b2ee97d26eb53f7c6acbeb34978a"
server_key_iv: "860799fd4370d3a2fce6ddc8c89e26d1"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.302183 16540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:50.303016 16554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.303216 16540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:50.303318 16540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "af631e8eca694aa9acf38735d0b5fb77"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "3698b2ee97d26eb53f7c6acbeb34978a"
server_key_iv: "860799fd4370d3a2fce6ddc8c89e26d1"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.303407 16540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:50.330226 16540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:50.330583 16540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:50.330745 16540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:50.331002 16540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:50.331352 16540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:50.331409 16540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.331454 16540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:50.331487 16540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.337872 16540 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:41827
I20260928 12:26:50.337918 16667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:41827 every 8 connection(s)
I20260928 12:26:50.338281 16540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:50.339284 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16540
I20260928 12:26:50.339396 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:50.339628 7245 external_mini_cluster.cc:1428] Setting key 1cb298c4bdf8449f155640e1c11ebda0
I20260928 12:26:50.340574 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35597
--builtin_ntp_servers=127.7.19.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 12:26:50.344430 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.339110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:35765 (local address 127.7.19.126:35597)
0928 12:26:50.339190 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.339193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.339464 (+ 271us) server_negotiation.cc:407] Connection header received
0928 12:26:50.340198 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.340199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.340213 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.340263 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.341175 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.341894 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.342862 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.343113 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.343204 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.344129 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:50.344131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:50.344131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:50.344136 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:50.344150 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:50.344154 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:50.344201 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:50.344312 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:26:50.344346 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:26:50.344656 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.339054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35597 (local address 127.7.19.65:35765)
0928 12:26:50.339307 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:50.339359 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:50.340079 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:50.340353 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:50.340358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:50.340370 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:50.341066 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:50.341072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.342003 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.342004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:50.342780 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:50.342782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.343209 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.343211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:50.343366 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.343963 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:50.343964 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:50.343994 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:50.344017 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:50.344213 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:50.344218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:50.344221 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:50.344310 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:26:50.344419 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":127,"threads_started":1}
I20260928 12:26:50.345172 16668 heartbeater.cc:347] Connected to a master server at 127.7.19.126:35597
I20260928 12:26:50.345263 16668 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:50.345464 16668 heartbeater.cc:511] Master 127.7.19.126:35597 requested a full tablet report, sending...
I20260928 12:26:50.346052 16480 ts_manager.cc:194] Registered new tserver with Master: af631e8eca694aa9acf38735d0b5fb77 (127.7.19.65:41827)
I20260928 12:26:50.347043 16480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.65:35765
W20260928 12:26:50.422572 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:50.422776 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:50.422801 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:50.423890 16671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:50.423924 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:50.423947 16671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:50.425312 16671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:50.425498 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:50.425724 16671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:50.430792 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.430843 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.430843 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.431119 16671 server_base.cc:1034] running on GCE node
I20260928 12:26:50.431361 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:50.431612 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:50.432735 16671 hybrid_clock.cc:648] HybridClock initialized: now 1790598410432723 us; error 28 us; skew 500 ppm
I20260928 12:26:50.434125 16671 webserver.cc:460] Webserver started at http://127.7.19.66:35099/ using document root <none> and password file <none>
I20260928 12:26:50.434414 16671 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:50.434487 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:50.434648 16671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:50.435791 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "daf6ae17fc10448582b2ea8161c375e6"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "0c356548ba1e764260d9be5175d9239c"
server_key_iv: "67e09bfcf2abec4e8332766e129caf59"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.436147 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "daf6ae17fc10448582b2ea8161c375e6"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "0c356548ba1e764260d9be5175d9239c"
server_key_iv: "67e09bfcf2abec4e8332766e129caf59"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.437829 16671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:26:50.438558 16685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.438723 16671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:50.438823 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "daf6ae17fc10448582b2ea8161c375e6"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "0c356548ba1e764260d9be5175d9239c"
server_key_iv: "67e09bfcf2abec4e8332766e129caf59"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.438917 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:50.459775 16671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:50.460127 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:50.460309 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:50.460565 16671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:50.460975 16671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:50.461033 16671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.461077 16671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:50.461129 16671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.467362 16671 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:35397
I20260928 12:26:50.467408 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:35397 every 8 connection(s)
I20260928 12:26:50.467747 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:50.473316 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.468522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:42693 (local address 127.7.19.126:35597)
0928 12:26:50.468595 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.468597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.468920 (+ 323us) server_negotiation.cc:407] Connection header received
0928 12:26:50.469474 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.469475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.469487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.469541 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.470239 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.470803 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.471803 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.471995 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.472074 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.473001 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:50.473003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:50.473004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:50.473008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:50.473023 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:50.473027 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:50.473099 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:50.473201 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:26:50.473235 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:50.473543 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.468527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35597 (local address 127.7.19.66:42693)
0928 12:26:50.468763 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:50.468812 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:50.469360 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:50.469613 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:50.469617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:50.469627 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:50.470128 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:50.470132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.470907 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.470908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:50.471699 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:50.471700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.472081 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.472082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:50.472219 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.472847 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:50.472847 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:50.472874 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:50.472897 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:50.473101 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:50.473107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:50.473110 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:50.473205 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 12:26:50.473316 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260928 12:26:50.474002 16799 heartbeater.cc:347] Connected to a master server at 127.7.19.126:35597
I20260928 12:26:50.474078 16799 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:50.474274 16799 heartbeater.cc:511] Master 127.7.19.126:35597 requested a full tablet report, sending...
I20260928 12:26:50.474648 16480 ts_manager.cc:194] Registered new tserver with Master: daf6ae17fc10448582b2ea8161c375e6 (127.7.19.66:35397)
I20260928 12:26:50.475026 16480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.66:42693
I20260928 12:26:50.475771 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16671
I20260928 12:26:50.475875 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:50.476085 7245 external_mini_cluster.cc:1428] Setting key 261f4f6290345c684af3947b5ff309b6
I20260928 12:26:50.476974 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35597
--builtin_ntp_servers=127.7.19.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 12:26:50.557204 16802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:50.557430 16802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:50.557451 16802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:50.558646 16802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:50.558689 16802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:50.558713 16802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:50.560051 16802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:35597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:50.560298 16802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:50.560597 16802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:50.565999 16808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.566107 16810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.566210 16807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.566186 16802 server_base.cc:1034] running on GCE node
I20260928 12:26:50.566448 16802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:50.566677 16802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:50.567864 16802 hybrid_clock.cc:648] HybridClock initialized: now 1790598410567863 us; error 42 us; skew 500 ppm
I20260928 12:26:50.569227 16802 webserver.cc:460] Webserver started at http://127.7.19.67:38307/ using document root <none> and password file <none>
I20260928 12:26:50.569465 16802 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:50.569512 16802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:50.569670 16802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:50.570760 16802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "0c5d2c8b1df343118bb5f146b56003f5"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "577b4d177866d8b40dc2f40f33a09b00"
server_key_iv: "aec3c5c3144b00a7ff9e839abc543a2c"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.571107 16802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "0c5d2c8b1df343118bb5f146b56003f5"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "577b4d177866d8b40dc2f40f33a09b00"
server_key_iv: "aec3c5c3144b00a7ff9e839abc543a2c"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.572402 16802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:50.573210 16816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.573411 16802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:26:50.573508 16802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "0c5d2c8b1df343118bb5f146b56003f5"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "577b4d177866d8b40dc2f40f33a09b00"
server_key_iv: "aec3c5c3144b00a7ff9e839abc543a2c"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.573586 16802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:50.596765 16802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:50.597093 16802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:50.597260 16802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:50.597493 16802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:50.597817 16802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:50.597870 16802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.597915 16802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:50.597941 16802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.603660 16802 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:40979
I20260928 12:26:50.603703 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:40979 every 8 connection(s)
I20260928 12:26:50.604050 16802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:50.609578 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.604877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:55961 (local address 127.7.19.126:35597)
0928 12:26:50.604947 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.604949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.605208 (+ 259us) server_negotiation.cc:407] Connection header received
0928 12:26:50.605759 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.605760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.605774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.605825 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.606541 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.607062 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.608080 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.608288 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.608374 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.609257 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:50.609258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:50.609259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:50.609264 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:50.609286 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:50.609291 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:50.609357 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:50.609459 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:26:50.609497 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:26:50.609802 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.604889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:35597 (local address 127.7.19.67:55961)
0928 12:26:50.605084 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:50.605118 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:50.605640 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:50.605896 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:50.605900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:50.605908 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:50.606407 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:50.606411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.607160 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.607161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:50.607957 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:50.607960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:50.608383 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:50.608384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:50.608516 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:50.609110 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:50.609111 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:50.609141 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:50.609163 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:50.609356 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:50.609362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:50.609365 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:50.609447 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:26:50.609550 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 12:26:50.610306 16930 heartbeater.cc:347] Connected to a master server at 127.7.19.126:35597
I20260928 12:26:50.610399 16930 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:50.610576 16930 heartbeater.cc:511] Master 127.7.19.126:35597 requested a full tablet report, sending...
I20260928 12:26:50.610939 16480 ts_manager.cc:194] Registered new tserver with Master: 0c5d2c8b1df343118bb5f146b56003f5 (127.7.19.67:40979)
I20260928 12:26:50.611267 16480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.67:55961
I20260928 12:26:50.611821 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16802
I20260928 12:26:50.611914 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:50.612128 7245 external_mini_cluster.cc:1428] Setting key 7d51673d524cf29e27e8de25198ab12a
I20260928 12:26:50.612932 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:26:50.614820 16527 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:50.614309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.7.19.126:35597)
0928 12:26:50.614374 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.614376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.614479 (+ 103us) server_negotiation.cc:407] Connection header received
0928 12:26:50.614555 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.614557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.614568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.614622 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.614744 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57662: BlockingRecv error: recv got EOF from 127.0.0.1:57662 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:26:50.614907 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16540
I20260928 12:26:50.620316 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16671
I20260928 12:26:50.624581 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16802
I20260928 12:26:50.628610 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16449
2026-09-28T12:26:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (600 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T12:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:50Z Disabled control of system clock
I20260928 12:26:50.655175 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45477
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:26:50.741247 16941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:50.741447 16941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:50.741468 16941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:50.742523 16941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:50.742558 16941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:50.742569 16941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:50.742583 16941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:50.742594 16941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:50.743842 16941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:45477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:50.744037 16941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:50.744264 16941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:50.749048 16949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.749059 16947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.749109 16946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.749282 16941 server_base.cc:1034] running on GCE node
I20260928 12:26:50.749555 16941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:50.749840 16941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:50.751006 16941 hybrid_clock.cc:648] HybridClock initialized: now 1790598410750984 us; error 33 us; skew 500 ppm
I20260928 12:26:50.752214 16941 webserver.cc:460] Webserver started at http://127.7.19.126:44507/ using document root <none> and password file <none>
I20260928 12:26:50.752497 16941 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:50.752571 16941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:50.752792 16941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:50.753912 16941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "59e29e55fd5b473798d8feac2133e1dc"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "6d9d3ffe506a298eba7023fdf1903b8f"
server_key_iv: "26007780551e97d09775650470974cb4"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.754266 16941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "59e29e55fd5b473798d8feac2133e1dc"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "6d9d3ffe506a298eba7023fdf1903b8f"
server_key_iv: "26007780551e97d09775650470974cb4"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.755574 16941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:50.756253 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.756440 16941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:50.756544 16941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "59e29e55fd5b473798d8feac2133e1dc"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "6d9d3ffe506a298eba7023fdf1903b8f"
server_key_iv: "26007780551e97d09775650470974cb4"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.756618 16941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:50.759919 16941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:50.760176 16941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:50.760322 16941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:50.764031 16941 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:45477
I20260928 12:26:50.764142 17007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:45477 every 8 connection(s)
I20260928 12:26:50.764582 16941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:50.765452 17008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:50.768091 17008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc: Bootstrap starting.
I20260928 12:26:50.768605 17008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:50.768970 17008 log.cc:824] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:50.769352 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 16941
I20260928 12:26:50.769459 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:50.769658 7245 external_mini_cluster.cc:1428] Setting key 47b715d47a4003a4905a09d7dbba11a5
I20260928 12:26:50.769690 17008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc: No bootstrap required, opened a new log
I20260928 12:26:50.771507 17008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e29e55fd5b473798d8feac2133e1dc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45477 } }
I20260928 12:26:50.771672 17008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:50.771711 17008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e29e55fd5b473798d8feac2133e1dc, State: Initialized, Role: FOLLOWER
I20260928 12:26:50.771847 17008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e29e55fd5b473798d8feac2133e1dc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45477 } }
I20260928 12:26:50.771970 17008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:50.772014 17008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:50.772063 17008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:50.772818 17008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e29e55fd5b473798d8feac2133e1dc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45477 } }
I20260928 12:26:50.772877 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.770198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.7.19.126:45477)
0928 12:26:50.770396 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.770403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.770428 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:26:50.771492 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.771499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.771538 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.771654 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.772390 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:50.772394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:50.772394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:50.772398 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:50.772434 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:50.772444 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:50.772500 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:50.772588 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 12:26:50.772690 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 12:26:50.772938 17008 leader_election.cc:302] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e29e55fd5b473798d8feac2133e1dc; no voters:
I20260928 12:26:50.773124 17008 leader_election.cc:288] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:50.773383 17008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:50.773401 17013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:50.773509 17013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [term 1 LEADER]: Becoming Leader. State: Replica: 59e29e55fd5b473798d8feac2133e1dc, State: Running, Role: LEADER
I20260928 12:26:50.773619 17013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e29e55fd5b473798d8feac2133e1dc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45477 } }
I20260928 12:26:50.773967 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e29e55fd5b473798d8feac2133e1dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e29e55fd5b473798d8feac2133e1dc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45477 } } }
I20260928 12:26:50.774055 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:50.774261 17015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e29e55fd5b473798d8feac2133e1dc. Latest consensus state: current_term: 1 leader_uuid: "59e29e55fd5b473798d8feac2133e1dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e29e55fd5b473798d8feac2133e1dc" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45477 } } }
I20260928 12:26:50.774343 17015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:50.774586 17022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:50.775092 17022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:50.776625 17022 catalog_manager.cc:1347] Generated new cluster ID: 97490d5de8ab4cb2b41b65772b8f0ee5
I20260928 12:26:50.776695 17022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:50.792704 17022 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:50.792909 17022 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:50.804418 17022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e29e55fd5b473798d8feac2133e1dc: Generated new TSK 0
I20260928 12:26:50.804651 17022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:26:50.807548 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45477
--builtin_ntp_servers=127.7.19.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:26:50.889716 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:50.889946 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:50.889986 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:50.891170 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:50.891235 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:50.891279 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:50.892782 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:50.893038 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:50.893323 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:50.899235 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.899305 17032 server_base.cc:1034] running on GCE node
W20260928 12:26:50.899255 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:50.899246 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:50.899631 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:50.899893 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:50.901046 17032 hybrid_clock.cc:648] HybridClock initialized: now 1790598410901034 us; error 33 us; skew 500 ppm
I20260928 12:26:50.902421 17032 webserver.cc:460] Webserver started at http://127.7.19.65:46803/ using document root <none> and password file <none>
I20260928 12:26:50.902661 17032 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:50.902717 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:50.902851 17032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:50.904209 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "645621ec20b64586a1e9732f61626cda"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "38666c47f5796fd5593461797f26a4e0"
server_key_iv: "961d77dfbc882d627c8d1119bdd12896"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.904585 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "645621ec20b64586a1e9732f61626cda"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "38666c47f5796fd5593461797f26a4e0"
server_key_iv: "961d77dfbc882d627c8d1119bdd12896"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.906147 17032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:50.907037 17046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.907232 17032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:50.907325 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "645621ec20b64586a1e9732f61626cda"
format_stamp: "Formatted at 2026-09-28 12:26:50 on dist-test-slave-cb7x"
server_key: "38666c47f5796fd5593461797f26a4e0"
server_key_iv: "961d77dfbc882d627c8d1119bdd12896"
server_key_version: "encryptionkey@0"
I20260928 12:26:50.907382 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:50.910552 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:50.910749 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:50.910838 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:50.910980 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:50.911198 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:50.911226 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.911244 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:50.911257 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:50.916527 17032 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:42317
I20260928 12:26:50.916571 17159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:42317 every 8 connection(s)
I20260928 12:26:50.916886 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:50.919605 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.917539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:37693 (local address 127.7.19.126:45477)
0928 12:26:50.917579 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:50.917581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:50.917819 (+ 238us) server_negotiation.cc:407] Connection header received
0928 12:26:50.918224 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:50.918225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:50.918229 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:50.918289 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:50.919423 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:50.919424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:50.919424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:50.919428 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:50.919442 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:50.919446 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:50.919494 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:50.919539 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:26:50.919555 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:26:50.919787 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:50.917520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45477 (local address 127.7.19.65:37693)
0928 12:26:50.917697 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:50.917734 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:50.918150 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:50.918340 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:50.918343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:50.918350 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:50.919277 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:50.919278 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:50.919303 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:50.919330 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:50.919492 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:50.919494 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:50.919497 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:50.919545 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:26:50.919595 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260928 12:26:50.920266 17160 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45477
I20260928 12:26:50.920363 17160 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:50.920511 17160 heartbeater.cc:511] Master 127.7.19.126:45477 requested a full tablet report, sending...
I20260928 12:26:50.920987 16972 ts_manager.cc:194] Registered new tserver with Master: 645621ec20b64586a1e9732f61626cda (127.7.19.65:42317)
I20260928 12:26:50.922207 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17032
I20260928 12:26:50.922281 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:50.922423 7245 external_mini_cluster.cc:1428] Setting key 124c466ddf5345ff731e4b53550c8eca
I20260928 12:26:50.923215 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45477
--builtin_ntp_servers=127.7.19.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:26:51.003786 17163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:51.003988 17163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:51.004009 17163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:51.005110 17163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:51.005144 17163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:51.005169 17163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:51.006393 17163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:51.006572 17163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:51.006798 17163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:51.011790 17168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:51.011763 17171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:51.011765 17169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:51.011978 17163 server_base.cc:1034] running on GCE node
I20260928 12:26:51.012219 17163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:51.012454 17163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:51.013613 17163 hybrid_clock.cc:648] HybridClock initialized: now 1790598411013607 us; error 37 us; skew 500 ppm
I20260928 12:26:51.014798 17163 webserver.cc:460] Webserver started at http://127.7.19.66:36407/ using document root <none> and password file <none>
I20260928 12:26:51.015044 17163 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:51.015112 17163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:51.015246 17163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:51.016283 17163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "168f358ac3114695b3e3408b7bd3be23"
format_stamp: "Formatted at 2026-09-28 12:26:51 on dist-test-slave-cb7x"
server_key: "4e9294588701525894efda725f2ea961"
server_key_iv: "491c2c1ab1a74f3c7996c2994364a680"
server_key_version: "encryptionkey@0"
I20260928 12:26:51.016656 17163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "168f358ac3114695b3e3408b7bd3be23"
format_stamp: "Formatted at 2026-09-28 12:26:51 on dist-test-slave-cb7x"
server_key: "4e9294588701525894efda725f2ea961"
server_key_iv: "491c2c1ab1a74f3c7996c2994364a680"
server_key_version: "encryptionkey@0"
I20260928 12:26:51.018129 17163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:26:51.018826 17177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:51.018981 17163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:26:51.019075 17163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "168f358ac3114695b3e3408b7bd3be23"
format_stamp: "Formatted at 2026-09-28 12:26:51 on dist-test-slave-cb7x"
server_key: "4e9294588701525894efda725f2ea961"
server_key_iv: "491c2c1ab1a74f3c7996c2994364a680"
server_key_version: "encryptionkey@0"
I20260928 12:26:51.019161 17163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:51.022473 17163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:51.022722 17163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:51.022858 17163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:51.023077 17163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:51.023366 17163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:51.023425 17163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:51.023464 17163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:51.023492 17163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:51.029155 17163 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:40899
I20260928 12:26:51.029189 17290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:40899 every 8 connection(s)
I20260928 12:26:51.029511 17163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:26:51.032317 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:51.030210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49567 (local address 127.7.19.126:45477)
0928 12:26:51.030259 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:51.030262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:51.030569 (+ 307us) server_negotiation.cc:407] Connection header received
0928 12:26:51.030986 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:51.030987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:51.030992 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:51.031047 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:51.032130 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:51.032132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:51.032132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:51.032136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:51.032150 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:51.032153 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:51.032200 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:51.032251 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 12:26:51.032263 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:26:51.032614 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:51.030203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45477 (local address 127.7.19.66:49567)
0928 12:26:51.030430 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:51.030461 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:51.030899 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:51.031140 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:51.031143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:51.031150 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:51.031997 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:51.031998 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:51.032023 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:51.032049 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:51.032198 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:51.032200 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:51.032202 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:51.032251 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:26:51.032295 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260928 12:26:51.033090 17291 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45477
I20260928 12:26:51.033164 17291 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:51.033308 17291 heartbeater.cc:511] Master 127.7.19.126:45477 requested a full tablet report, sending...
I20260928 12:26:51.033674 16972 ts_manager.cc:194] Registered new tserver with Master: 168f358ac3114695b3e3408b7bd3be23 (127.7.19.66:40899)
I20260928 12:26:51.037626 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17163
I20260928 12:26:51.037704 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:51.037844 7245 external_mini_cluster.cc:1428] Setting key 64b8be72ad2b7872bec5f0587504834b
I20260928 12:26:51.038726 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45477
--builtin_ntp_servers=127.7.19.84:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 12:26:51.118486 17294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:51.118716 17294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:51.118750 17294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:51.119911 17294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:51.119973 17294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:51.120018 17294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:51.121361 17294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:51.121625 17294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:51.121918 17294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:51.126873 17300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:51.126852 17302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:51.126997 17294 server_base.cc:1034] running on GCE node
W20260928 12:26:51.126852 17299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:51.127233 17294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:51.127499 17294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:51.128639 17294 hybrid_clock.cc:648] HybridClock initialized: now 1790598411128616 us; error 39 us; skew 500 ppm
I20260928 12:26:51.129850 17294 webserver.cc:460] Webserver started at http://127.7.19.67:46429/ using document root <none> and password file <none>
I20260928 12:26:51.130080 17294 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:51.130125 17294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:51.130275 17294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:51.131321 17294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "9d91106adf4d49068e65b6ad941b3302"
format_stamp: "Formatted at 2026-09-28 12:26:51 on dist-test-slave-cb7x"
server_key: "0bd94b00aff562d862bcb5ffe2a6bbc6"
server_key_iv: "7a464522c54fb5b40a22b38c50f7c9a2"
server_key_version: "encryptionkey@0"
I20260928 12:26:51.131685 17294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "9d91106adf4d49068e65b6ad941b3302"
format_stamp: "Formatted at 2026-09-28 12:26:51 on dist-test-slave-cb7x"
server_key: "0bd94b00aff562d862bcb5ffe2a6bbc6"
server_key_iv: "7a464522c54fb5b40a22b38c50f7c9a2"
server_key_version: "encryptionkey@0"
I20260928 12:26:51.132997 17294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:51.133700 17308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:51.133872 17294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:51.133941 17294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "9d91106adf4d49068e65b6ad941b3302"
format_stamp: "Formatted at 2026-09-28 12:26:51 on dist-test-slave-cb7x"
server_key: "0bd94b00aff562d862bcb5ffe2a6bbc6"
server_key_iv: "7a464522c54fb5b40a22b38c50f7c9a2"
server_key_version: "encryptionkey@0"
I20260928 12:26:51.134029 17294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:51.137300 17294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:51.137547 17294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:51.137679 17294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:51.137871 17294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:51.138152 17294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:51.138221 17294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:51.138259 17294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:51.138288 17294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:51.143869 17294 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:44715
I20260928 12:26:51.143918 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:44715 every 8 connection(s)
I20260928 12:26:51.144197 17294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:26:51.146834 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:51.144879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:56273 (local address 127.7.19.126:45477)
0928 12:26:51.144926 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:51.144928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:51.145132 (+ 204us) server_negotiation.cc:407] Connection header received
0928 12:26:51.145539 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:51.145540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:51.145544 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:51.145625 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:51.146657 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:51.146659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:51.146659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:26:51.146663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:51.146677 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:26:51.146681 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:51.146728 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:51.146773 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 12:26:51.146784 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:51.147051 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:51.144833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45477 (local address 127.7.19.67:56273)
0928 12:26:51.145001 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:51.145040 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:51.145457 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:51.145671 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:51.145673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:51.145681 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:51.146512 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:26:51.146513 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:51.146539 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:26:51.146565 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:51.146725 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:51.146728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:51.146730 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:26:51.146778 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:26:51.146819 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260928 12:26:51.147547 17422 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45477
I20260928 12:26:51.147619 17422 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:51.147781 17422 heartbeater.cc:511] Master 127.7.19.126:45477 requested a full tablet report, sending...
I20260928 12:26:51.148299 16972 ts_manager.cc:194] Registered new tserver with Master: 9d91106adf4d49068e65b6ad941b3302 (127.7.19.67:44715)
I20260928 12:26:51.153081 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17294
I20260928 12:26:51.153185 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:51.153401 7245 external_mini_cluster.cc:1428] Setting key 21f3612a85df48f248969fd5c88c91ec
I20260928 12:26:51.154254 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 12:26:51.156097 17012 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:26:51.155609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.7.19.126:45477)
0928 12:26:51.155676 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:51.155678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:51.155801 (+ 123us) server_negotiation.cc:407] Connection header received
0928 12:26:51.155827 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:51.155828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:51.155832 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:51.155885 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:51.156021 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40274: BlockingRecv error: recv got EOF from 127.0.0.1:40274 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:26:51.156117 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17032
I20260928 12:26:51.160961 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17163
I20260928 12:26:51.165726 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17294
I20260928 12:26:51.169662 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 16941
2026-09-28T12:26:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (540 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:51 dist-test-slave-cb7x krb5kdc[17432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:51 dist-test-slave-cb7x krb5kdc[17432](info): set up 2 sockets
Sep 28 12:26:51 dist-test-slave-cb7x krb5kdc[17432](info): commencing operation
krb5kdc: starting...
W20260928 12:26:52.205493 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:26:52.319871 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46293
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:52.398653 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:52.398926 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:52.398968 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:52.400170 17445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:26:52.400244 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:52.400267 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:52.400288 17445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:26:52.400318 17445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:26:52.401885 17445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:52.402156 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:52.402384 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:26:52.407307 17445 server_base.cc:1034] running on GCE node
W20260928 12:26:52.407284 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:52.407328 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:52.407291 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:52.407737 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:52.408057 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:52.409230 17445 hybrid_clock.cc:648] HybridClock initialized: now 1790598412409197 us; error 48 us; skew 500 ppm
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:52.411522 17445 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:26:52.411918 17445 webserver.cc:460] Webserver started at http://127.7.19.126:42103/ using document root <none> and password file <none>
I20260928 12:26:52.412153 17445 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:52.412223 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:52.412374 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:52.413467 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "6b08c6d3d34044569d6236da85b40e58"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "346e6ec4106bb6d07d4057cdf80455ac"
server_key_iv: "4ad94f9bf50ac8eb32284e7eae31ef14"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.413836 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "6b08c6d3d34044569d6236da85b40e58"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "346e6ec4106bb6d07d4057cdf80455ac"
server_key_iv: "4ad94f9bf50ac8eb32284e7eae31ef14"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.415139 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:52.415927 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.416239 17445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:52.416371 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "6b08c6d3d34044569d6236da85b40e58"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "346e6ec4106bb6d07d4057cdf80455ac"
server_key_iv: "4ad94f9bf50ac8eb32284e7eae31ef14"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.416447 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:52.436918 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:52.438750 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:52.438900 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:52.442905 17445 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:46293
I20260928 12:26:52.442960 17512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:46293 every 8 connection(s)
I20260928 12:26:52.443305 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:26:52.444064 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17445
I20260928 12:26:52.444167 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:26:52.444283 17513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:52.444461 7245 external_mini_cluster.cc:1428] Setting key 1e4444ee3a419cfa576a7de7d22e7f86
I20260928 12:26:52.447587 17513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: Bootstrap starting.
I20260928 12:26:52.448249 17513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:52.448537 17513 log.cc:824] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:52.449221 17513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: No bootstrap required, opened a new log
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:52.451725 17513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b08c6d3d34044569d6236da85b40e58" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46293 } }
I20260928 12:26:52.451874 17513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:52.451920 17513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b08c6d3d34044569d6236da85b40e58, State: Initialized, Role: FOLLOWER
I20260928 12:26:52.452042 17513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b08c6d3d34044569d6236da85b40e58" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46293 } }
I20260928 12:26:52.452142 17513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:26:52.452188 17513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:26:52.452235 17513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:52.452780 17513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b08c6d3d34044569d6236da85b40e58" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46293 } }
I20260928 12:26:52.452886 17513 leader_election.cc:302] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b08c6d3d34044569d6236da85b40e58; no voters:
I20260928 12:26:52.453037 17513 leader_election.cc:288] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:26:52.453073 17518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:52.453259 17518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [term 1 LEADER]: Becoming Leader. State: Replica: 6b08c6d3d34044569d6236da85b40e58, State: Running, Role: LEADER
I20260928 12:26:52.453318 17513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:26:52.453442 17518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b08c6d3d34044569d6236da85b40e58" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46293 } }
I20260928 12:26:52.453838 17518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b08c6d3d34044569d6236da85b40e58. Latest consensus state: current_term: 1 leader_uuid: "6b08c6d3d34044569d6236da85b40e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b08c6d3d34044569d6236da85b40e58" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46293 } } }
I20260928 12:26:52.453913 17518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [sys.catalog]: This master's current role is: LEADER
I20260928 12:26:52.454030 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b08c6d3d34044569d6236da85b40e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b08c6d3d34044569d6236da85b40e58" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46293 } } }
I20260928 12:26:52.454103 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58 [sys.catalog]: This master's current role is: LEADER
W20260928 12:26:52.455390 17534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:26:52.455485 17534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:26:52.455651 17523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:26:52.455783 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.444971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.7.19.126:46293)
0928 12:26:52.445169 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.445178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.445211 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:26:52.445882 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.445892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.445954 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.446104 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:52.446688 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.448034 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.448900 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.449326 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.449457 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.451830 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:52.451833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:52.451833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:52.451843 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:52.453871 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.454316 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.454320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.454320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.454427 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.454647 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.454648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.454649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.455002 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:52.455149 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.455406 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.455544 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260928 12:26:52.455837 17523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:26:52.457288 17523 catalog_manager.cc:1347] Generated new cluster ID: 12679f4a025147f5a30c8bb077dff5ea
I20260928 12:26:52.457327 17523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:26:52.464301 17523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:26:52.464823 17523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:26:52.467249 17523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: Generated new TSK 0
I20260928 12:26:52.467402 17523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:26:52.499974 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46293
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:52.579026 17539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:52.579218 17539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:52.579241 17539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:52.580302 17539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:52.580334 17539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:52.580359 17539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:26:52.582443 17539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:52.582618 17539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:52.582779 17539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:52.587638 17545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:52.587638 17547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:52.587869 17539 server_base.cc:1034] running on GCE node
W20260928 12:26:52.587868 17544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:52.588165 17539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:52.588402 17539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:52.589571 17539 hybrid_clock.cc:648] HybridClock initialized: now 1790598412589558 us; error 32 us; skew 500 ppm
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:52.592056 17539 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:26:52.592517 17539 webserver.cc:460] Webserver started at http://127.7.19.65:34009/ using document root <none> and password file <none>
I20260928 12:26:52.592799 17539 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:52.592867 17539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:52.593005 17539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:52.594027 17539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "9a53544834bd0319b7503e25da7be9c4"
server_key_iv: "abf88288c662b6b1335560bb867fd412"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.594419 17539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "9a53544834bd0319b7503e25da7be9c4"
server_key_iv: "abf88288c662b6b1335560bb867fd412"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.595801 17539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:26:52.596581 17554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.596769 17539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:26:52.596865 17539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "9a53544834bd0319b7503e25da7be9c4"
server_key_iv: "abf88288c662b6b1335560bb867fd412"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.596956 17539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:52.611116 17539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:52.613294 17539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:52.613451 17539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:52.613677 17539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:52.613976 17539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:52.614028 17539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.614070 17539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:52.614099 17539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.620081 17539 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:38513
I20260928 12:26:52.620146 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:38513 every 8 connection(s)
I20260928 12:26:52.620434 17539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:26:52.624523 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17539
I20260928 12:26:52.624625 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:26:52.624853 7245 external_mini_cluster.cc:1428] Setting key b0797e621e9729339d7a140ff051c3ee
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:52.629938 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.621074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46293 (local address 127.7.19.65:36719)
0928 12:26:52.621243 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:52.621275 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:52.621766 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:52.622022 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:52.622026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:52.622588 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:52.622764 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.622768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.623477 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.623479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.624263 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.624264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.624721 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.624722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.624835 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.625469 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:52.625479 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:52.626811 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:52.628572 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:52.628577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:52.628579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:52.628853 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:52.629103 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:52.629103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:52.629103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:52.629194 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:52.629466 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:52.629470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:52.629576 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 12:26:52.629657 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:26:52.629767 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 12:26:52.629947 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.621099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:36719 (local address 127.7.19.126:46293)
0928 12:26:52.621143 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.621145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.621361 (+ 216us) server_negotiation.cc:407] Connection header received
0928 12:26:52.621872 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.621873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.621886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.621956 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:52.622865 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.623380 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.624356 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.624581 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.624661 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.626906 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:52.626908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:52.626908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:52.626916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:52.628491 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.628963 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.628964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.628965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.629029 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.629257 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.629258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.629259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.629397 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:52.629471 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.629703 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.629793 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:26:52.630729 17668 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46293
I20260928 12:26:52.630829 17668 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:52.631035 17668 heartbeater.cc:511] Master 127.7.19.126:46293 requested a full tablet report, sending...
I20260928 12:26:52.631552 17477 ts_manager.cc:194] Registered new tserver with Master: 176e6ec0dd4d4f60a1d9b68615ace0e9 (127.7.19.65:38513)
I20260928 12:26:52.632148 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:36719
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:26:52.650966 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46293
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:52.732952 17673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:52.733158 17673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:52.733196 17673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:52.734295 17673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:52.734356 17673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:52.734400 17673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:26:52.735800 17673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:52.736052 17673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:52.736277 17673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:52.741020 17681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:52.741021 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:52.741214 17679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:52.741245 17673 server_base.cc:1034] running on GCE node
I20260928 12:26:52.741428 17673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:52.741631 17673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:52.742865 17673 hybrid_clock.cc:648] HybridClock initialized: now 1790598412742846 us; error 38 us; skew 500 ppm
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:52.745200 17673 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:26:52.745613 17673 webserver.cc:460] Webserver started at http://127.7.19.66:36335/ using document root <none> and password file <none>
I20260928 12:26:52.745800 17673 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:52.745837 17673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:52.745920 17673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:52.746901 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "878265bdd8054227a830ebf217b77ecd"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "ecb4fbe609f31ede3fd7522fa38be034"
server_key_iv: "5ba84cb27cddc1d431ac7be9c38d9fc0"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.747205 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "878265bdd8054227a830ebf217b77ecd"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "ecb4fbe609f31ede3fd7522fa38be034"
server_key_iv: "5ba84cb27cddc1d431ac7be9c38d9fc0"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.748374 17673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:26:52.749095 17688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.749257 17673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.749354 17673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "878265bdd8054227a830ebf217b77ecd"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "ecb4fbe609f31ede3fd7522fa38be034"
server_key_iv: "5ba84cb27cddc1d431ac7be9c38d9fc0"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.749432 17673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:52.760910 17673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:52.762533 17673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:52.762689 17673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:52.762900 17673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:52.763188 17673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:52.763239 17673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.763273 17673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:52.763299 17673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.768972 17673 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:32889
I20260928 12:26:52.769006 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:32889 every 8 connection(s)
I20260928 12:26:52.769304 17673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:52.775092 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17673
I20260928 12:26:52.775202 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:26:52.775400 7245 external_mini_cluster.cc:1428] Setting key c69ed1cc23d934f415fd780589a1ca1e
I20260928 12:26:52.778133 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.770002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:35141 (local address 127.7.19.126:46293)
0928 12:26:52.770069 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.770070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.770213 (+ 143us) server_negotiation.cc:407] Connection header received
0928 12:26:52.770659 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.770660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.770669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.770717 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:52.771536 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.772007 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.772941 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.773135 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.773201 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.775184 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:52.775186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:52.775186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:52.775194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:52.776635 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.777140 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.777140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.777141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.777210 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.777498 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.777499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.777499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.777614 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:52.777687 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.777963 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.778044 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:26:52.778308 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.769959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46293 (local address 127.7.19.66:35141)
0928 12:26:52.770099 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:52.770137 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:52.770583 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:52.770772 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:52.770775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:52.771292 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:52.771467 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.771472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.772078 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.772079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.772846 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.772848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.773212 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.773212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.773306 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.773698 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:52.773705 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:52.775075 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:52.776740 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:52.776747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:52.776749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:52.777071 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:52.777314 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:52.777316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:52.777316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:52.777426 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:52.777692 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:52.777697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:52.777823 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 12:26:52.777940 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 12:26:52.778088 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 12:26:52.778690 17802 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46293
I20260928 12:26:52.778795 17802 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:52.778980 17802 heartbeater.cc:511] Master 127.7.19.126:46293 requested a full tablet report, sending...
I20260928 12:26:52.779331 17477 ts_manager.cc:194] Registered new tserver with Master: 878265bdd8054227a830ebf217b77ecd (127.7.19.66:32889)
I20260928 12:26:52.779872 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:35141
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:26:52.803593 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46293
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:26:52.886232 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:26:52.886518 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:26:52.886543 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:26:52.887691 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:26:52.887730 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:26:52.887756 17807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:26:52.889295 17807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:26:52.889519 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:26:52.889719 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:26:52.894714 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:26:52.894726 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:52.894953 17807 server_base.cc:1034] running on GCE node
W20260928 12:26:52.894723 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:26:52.895182 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:26:52.895390 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:26:52.896550 17807 hybrid_clock.cc:648] HybridClock initialized: now 1790598412896528 us; error 36 us; skew 500 ppm
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:26:52.899114 17807 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:26:52.899518 17807 webserver.cc:460] Webserver started at http://127.7.19.67:40643/ using document root <none> and password file <none>
I20260928 12:26:52.899775 17807 fs_manager.cc:359] Metadata directory not provided
I20260928 12:26:52.899848 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:26:52.900008 17807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:26:52.901147 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "29ee768ab1a34dab838eb25f2ec7464d"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "a38465c8e8fa0be798f9f93d58468033"
server_key_iv: "8e3b3da9842f9f80899b8c4f406cb739"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.901523 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "29ee768ab1a34dab838eb25f2ec7464d"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "a38465c8e8fa0be798f9f93d58468033"
server_key_iv: "8e3b3da9842f9f80899b8c4f406cb739"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.902799 17807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:26:52.903554 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.903720 17807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:26:52.903820 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "29ee768ab1a34dab838eb25f2ec7464d"
format_stamp: "Formatted at 2026-09-28 12:26:52 on dist-test-slave-cb7x"
server_key: "a38465c8e8fa0be798f9f93d58468033"
server_key_iv: "8e3b3da9842f9f80899b8c4f406cb739"
server_key_version: "encryptionkey@0"
I20260928 12:26:52.903901 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:26:52.912276 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:26:52.914076 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:26:52.914244 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:26:52.914455 17807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:26:52.914757 17807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:26:52.914812 17807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.914855 17807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:26:52.914882 17807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:26:52.920594 17807 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:43423
I20260928 12:26:52.920661 17935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:43423 every 8 connection(s)
I20260928 12:26:52.920998 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:26:52 dist-test-slave-cb7x krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598412, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:26:52.928306 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 17807
I20260928 12:26:52.928390 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:26:52.928617 7245 external_mini_cluster.cc:1428] Setting key 89ae4fe2c2d021cdb2d3d317726caa19
I20260928 12:26:52.929944 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.921693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:40281 (local address 127.7.19.126:46293)
0928 12:26:52.921758 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.921760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.921968 (+ 208us) server_negotiation.cc:407] Connection header received
0928 12:26:52.922444 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.922444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.922454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.922508 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:52.923381 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.923871 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.924949 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.925127 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.925179 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.927209 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:52.927210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:52.927210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:52.927219 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:52.928477 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.928969 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.928971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.928971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.929040 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.929294 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.929295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.929296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.929420 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:52.929503 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.929766 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.929857 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:26:52.930136 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.921674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46293 (local address 127.7.19.67:40281)
0928 12:26:52.921843 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:52.921884 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:52.922363 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:52.922559 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:52.922563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:52.923119 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:26:52.923304 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.923308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.923959 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.923960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.924876 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.924879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.925200 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.925201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.925313 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.925791 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:26:52.925799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:26:52.927122 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:26:52.928539 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:52.928546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:52.928548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:52.928867 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:52.929111 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:26:52.929112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:26:52.929112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:26:52.929224 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:26:52.929493 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:26:52.929498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:26:52.929639 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:26:52.929746 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 12:26:52.929899 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260928 12:26:52.930605 17936 heartbeater.cc:347] Connected to a master server at 127.7.19.126:46293
I20260928 12:26:52.930708 17936 heartbeater.cc:464] Registering TS with master...
I20260928 12:26:52.930873 17936 heartbeater.cc:511] Master 127.7.19.126:46293 requested a full tablet report, sending...
I20260928 12:26:52.931247 17477 ts_manager.cc:194] Registered new tserver with Master: 29ee768ab1a34dab838eb25f2ec7464d (127.7.19.67:43423)
I20260928 12:26:52.931596 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:40281
I20260928 12:26:52.940348 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:26:52.947402 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.941729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.7.19.126:46293)
0928 12:26:52.941789 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.941790 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.941877 (+ 87us) server_negotiation.cc:407] Connection header received
0928 12:26:52.941930 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.941930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.941939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.941996 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:26:52.942584 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.943049 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.943787 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.943967 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.944031 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.945198 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:26:52.945199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:26:52.945199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:26:52.945206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:26:52.946463 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.946752 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.946753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.946753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.946800 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:26:52.946959 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:26:52.946959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:26:52.946959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:26:52.947084 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:26:52.947156 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.947257 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.947311 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:26:52.948904 17477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:26:52.950459 17477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:26:52.957304 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.952839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.7.19.66:32889)
0928 12:26:52.953005 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.953011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.953171 (+ 160us) server_negotiation.cc:407] Connection header received
0928 12:26:52.953350 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.953352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.953368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.953492 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:52.953683 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.954844 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.956305 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.956861 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.956957 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.957017 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.957165 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.957206 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260928 12:26:52.957319 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.952721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:32889 (local address 127.0.0.1:38722)
0928 12:26:52.953003 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:52.953042 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:52.953113 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:52.953562 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:52.953563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:52.953564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:52.953607 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.953608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.954974 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.954976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.956207 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.956208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.956959 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.956960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.957035 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.957049 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:26:52.957149 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 12:26:52.957203 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":6,"thread_start_us":57,"threads_started":1}
I20260928 12:26:52.957497 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.953103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.7.19.65:38513)
0928 12:26:52.953395 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.953400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.953410 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:26:52.953461 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.953463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.953479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.953551 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:52.953694 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.954965 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.956293 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.956984 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.957042 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.957088 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.957359 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.957402 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":73,"threads_started":1}
I20260928 12:26:52.957510 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.952721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:43423 (local address 127.0.0.1:42196)
0928 12:26:52.953002 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:52.953043 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:52.953113 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:52.953486 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:52.953489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:52.953491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:52.953548 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.953550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.954710 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.954711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.956060 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.956062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.957074 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.957075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.957136 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.957140 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:52.957190 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:26:52.957253 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":80,"thread_start_us":54,"threads_started":1}
I20260928 12:26:52.957597 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.952924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38513 (local address 127.0.0.1:54138)
0928 12:26:52.953108 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:52.953120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:52.953200 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:52.953560 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:52.953561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:52.953563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:52.953611 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.953613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.955058 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.955058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.956205 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:52.956208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.957115 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.957115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:52.957175 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.957179 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:52.957240 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:26:52.957282 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260928 12:26:52.957794 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:52.952830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.7.19.67:43423)
0928 12:26:52.953268 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:52.953273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:52.953281 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:26:52.953317 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:52.953319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:52.953334 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:52.953407 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:52.953625 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.954605 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.956145 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:52.956912 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:52.956983 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:52.957039 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:52.957359 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 12:26:52.957705 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":86,"threads_started":1}
I20260928 12:26:52.958341 17870 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea775984f7d4755aead01d5c41d8d59 (DEFAULT_TABLE table=test-table [id=c72bdb800f0046ab96a2294e1bbcba6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:52.958338 17602 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea775984f7d4755aead01d5c41d8d59 (DEFAULT_TABLE table=test-table [id=c72bdb800f0046ab96a2294e1bbcba6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:52.958663 17870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea775984f7d4755aead01d5c41d8d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:52.958678 17602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea775984f7d4755aead01d5c41d8d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:52.961092 17736 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea775984f7d4755aead01d5c41d8d59 (DEFAULT_TABLE table=test-table [id=c72bdb800f0046ab96a2294e1bbcba6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:26:52.961409 17736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea775984f7d4755aead01d5c41d8d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:26:52.961297 17950 tablet_bootstrap.cc:492] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: Bootstrap starting.
I20260928 12:26:52.961941 17950 tablet_bootstrap.cc:654] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:52.962327 17950 log.cc:824] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:52.962617 17951 tablet_bootstrap.cc:492] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Bootstrap starting.
I20260928 12:26:52.963074 17950 tablet_bootstrap.cc:492] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: No bootstrap required, opened a new log
I20260928 12:26:52.963132 17950 ts_tablet_manager.cc:1397] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:52.963145 17951 tablet_bootstrap.cc:654] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:52.963534 17951 log.cc:824] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:52.964254 17951 tablet_bootstrap.cc:492] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: No bootstrap required, opened a new log
I20260928 12:26:52.964322 17951 ts_tablet_manager.cc:1397] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:26:52.964514 17954 tablet_bootstrap.cc:492] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: Bootstrap starting.
I20260928 12:26:52.965133 17954 tablet_bootstrap.cc:654] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: Neither blocks nor log segments found. Creating new log.
I20260928 12:26:52.965289 17950 raft_consensus.cc:348] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:52.965497 17950 raft_consensus.cc:374] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:52.965574 17950 raft_consensus.cc:729] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176e6ec0dd4d4f60a1d9b68615ace0e9, State: Initialized, Role: FOLLOWER
I20260928 12:26:52.965687 17954 log.cc:824] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:26:52.965703 17950 consensus_queue.cc:260] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:52.966004 17950 ts_tablet_manager.cc:1428] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:26:52.965958 17951 raft_consensus.cc:348] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:52.966121 17951 raft_consensus.cc:374] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:52.966142 17668 heartbeater.cc:503] Master 127.7.19.126:46293 was elected leader, sending a full tablet report...
I20260928 12:26:52.966168 17951 raft_consensus.cc:729] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ee768ab1a34dab838eb25f2ec7464d, State: Initialized, Role: FOLLOWER
I20260928 12:26:52.966281 17951 consensus_queue.cc:260] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:52.966552 17951 ts_tablet_manager.cc:1428] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:26:52.966712 17936 heartbeater.cc:503] Master 127.7.19.126:46293 was elected leader, sending a full tablet report...
I20260928 12:26:52.966984 17954 tablet_bootstrap.cc:492] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: No bootstrap required, opened a new log
I20260928 12:26:52.967198 17954 ts_tablet_manager.cc:1397] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:26:52.968853 17954 raft_consensus.cc:348] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:52.968971 17954 raft_consensus.cc:374] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:26:52.968992 17954 raft_consensus.cc:729] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878265bdd8054227a830ebf217b77ecd, State: Initialized, Role: FOLLOWER
I20260928 12:26:52.969061 17954 consensus_queue.cc:260] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:52.969298 17954 ts_tablet_manager.cc:1428] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:26:52.969324 17802 heartbeater.cc:503] Master 127.7.19.126:46293 was elected leader, sending a full tablet report...
W20260928 12:26:53.019887 17803 tablet.cc:2369] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:53.106953 17957 raft_consensus.cc:482] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:26:53.107070 17957 raft_consensus.cc:504] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:53.107379 17957 leader_election.cc:288] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 878265bdd8054227a830ebf217b77ecd (127.7.19.66:32889), 176e6ec0dd4d4f60a1d9b68615ace0e9 (127.7.19.65:38513)
I20260928 12:26:53.110214 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.107648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:48193 (local address 127.7.19.66:32889)
0928 12:26:53.107702 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:53.107704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:53.107716 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:26:53.107751 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:53.107752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:53.107764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:53.107833 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:53.108035 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.108508 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.109490 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.109857 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.109936 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:53.109947 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:53.110110 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 12:26:53.110145 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 12:26:53.110214 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.107578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:32889 (local address 127.7.19.67:48193)
0928 12:26:53.107635 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:53.107647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:53.107690 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:53.107851 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:53.107852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:53.107854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:53.107929 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:53.107931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.108589 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.108590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:53.109388 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:53.109390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.109953 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.109954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:53.110034 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:53.110037 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:26:53.110103 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:26:53.110141 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:26:53.110493 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.107784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:43439 (local address 127.7.19.65:38513)
0928 12:26:53.107822 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:53.107824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:53.108003 (+ 179us) server_negotiation.cc:407] Connection header received
0928 12:26:53.108030 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:53.108031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:53.108043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:53.108111 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:26:53.108270 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.108806 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.109710 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.110087 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.110170 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:53.110179 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:53.110361 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 12:26:53.110424 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:26:53.110574 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea775984f7d4755aead01d5c41d8d59" candidate_uuid: "29ee768ab1a34dab838eb25f2ec7464d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878265bdd8054227a830ebf217b77ecd" is_pre_election: true
I20260928 12:26:53.110724 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.107698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38513 (local address 127.7.19.67:43439)
0928 12:26:53.107893 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:53.107903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:26:53.107977 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:26:53.108126 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:26:53.108127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:26:53.108128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:26:53.108204 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:53.108205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.108885 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.108886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:53.109619 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:26:53.109621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.110171 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.110172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:26:53.110243 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:53.110247 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:26:53.110615 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 12:26:53.110656 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260928 12:26:53.110746 17756 raft_consensus.cc:2454] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29ee768ab1a34dab838eb25f2ec7464d in term 0.
I20260928 12:26:53.111013 17824 leader_election.cc:302] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ee768ab1a34dab838eb25f2ec7464d, 878265bdd8054227a830ebf217b77ecd; no voters:
I20260928 12:26:53.111135 17957 raft_consensus.cc:2781] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:26:53.111192 17957 raft_consensus.cc:482] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:26:53.111224 17957 raft_consensus.cc:3037] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:53.111754 17622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea775984f7d4755aead01d5c41d8d59" candidate_uuid: "29ee768ab1a34dab838eb25f2ec7464d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" is_pre_election: true
I20260928 12:26:53.111861 17957 raft_consensus.cc:504] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:53.111923 17622 raft_consensus.cc:2454] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29ee768ab1a34dab838eb25f2ec7464d in term 0.
I20260928 12:26:53.112006 17957 leader_election.cc:288] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [CANDIDATE]: Term 1 election: Requested vote from peers 878265bdd8054227a830ebf217b77ecd (127.7.19.66:32889), 176e6ec0dd4d4f60a1d9b68615ace0e9 (127.7.19.65:38513)
I20260928 12:26:53.112151 17756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea775984f7d4755aead01d5c41d8d59" candidate_uuid: "29ee768ab1a34dab838eb25f2ec7464d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878265bdd8054227a830ebf217b77ecd"
I20260928 12:26:53.112198 17622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea775984f7d4755aead01d5c41d8d59" candidate_uuid: "29ee768ab1a34dab838eb25f2ec7464d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9"
I20260928 12:26:53.112254 17756 raft_consensus.cc:3037] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:53.112294 17622 raft_consensus.cc:3037] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:26:53.113029 17622 raft_consensus.cc:2454] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ee768ab1a34dab838eb25f2ec7464d in term 1.
I20260928 12:26:53.113036 17756 raft_consensus.cc:2454] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ee768ab1a34dab838eb25f2ec7464d in term 1.
I20260928 12:26:53.113193 17824 leader_election.cc:302] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ee768ab1a34dab838eb25f2ec7464d, 878265bdd8054227a830ebf217b77ecd; no voters:
I20260928 12:26:53.113315 17957 raft_consensus.cc:2781] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:26:53.113497 17957 raft_consensus.cc:686] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 1 LEADER]: Becoming Leader. State: Replica: 29ee768ab1a34dab838eb25f2ec7464d, State: Running, Role: LEADER
I20260928 12:26:53.113591 17957 consensus_queue.cc:237] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } }
I20260928 12:26:53.114377 17476 catalog_manager.cc:5495] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d reported cstate change: term changed from 0 to 1, leader changed from <none> to 29ee768ab1a34dab838eb25f2ec7464d (127.7.19.67). New cstate: current_term: 1 leader_uuid: "29ee768ab1a34dab838eb25f2ec7464d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29ee768ab1a34dab838eb25f2ec7464d" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 43423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 } health_report { overall_health: UNKNOWN } } }
W20260928 12:26:53.121130 17669 tablet.cc:2369] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:26:53.137521 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.135226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.7.19.67:43423)
0928 12:26:53.135289 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 12:26:53.135290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:26:53.135299 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:26:53.135357 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:26:53.135358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:26:53.135375 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:26:53.135443 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:26:53.135554 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.136024 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.136776 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:26:53.136981 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:26:53.137032 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:26:53.137160 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:26:53.137283 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:26:53.137345 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:26:53.137376 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:26:53.137395 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:26:53.139354 17622 raft_consensus.cc:1261] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 1 FOLLOWER]: Refusing update from remote peer 29ee768ab1a34dab838eb25f2ec7464d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:53.139354 17756 raft_consensus.cc:1261] T 1ea775984f7d4755aead01d5c41d8d59 P 878265bdd8054227a830ebf217b77ecd [term 1 FOLLOWER]: Refusing update from remote peer 29ee768ab1a34dab838eb25f2ec7464d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:26:53.139721 17960 consensus_queue.cc:1035] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [LEADER]: Connected to new peer: Peer: permanent_uuid: "878265bdd8054227a830ebf217b77ecd" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:53.139869 17957 consensus_queue.cc:1035] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [LEADER]: Connected to new peer: Peer: permanent_uuid: "176e6ec0dd4d4f60a1d9b68615ace0e9" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:26:53.142127 17964 mvcc.cc:205] Tried to move back new op lower bound from 7334291100215799808 to 7334291100113846272. Current Snapshot: MvccSnapshot[applied={T|T < 7334291100215799808}]
I20260928 12:26:53.142933 17965 mvcc.cc:205] Tried to move back new op lower bound from 7334291100215799808 to 7334291100113846272. Current Snapshot: MvccSnapshot[applied={T|T < 7334291100215799808 or (T in {7334291100215799808})}]
I20260928 12:26:53.144853 17476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:53.144958 17476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:26:53.145673 17476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b08c6d3d34044569d6236da85b40e58: Sending DeleteTablet for 3 replicas of tablet 1ea775984f7d4755aead01d5c41d8d59
I20260928 12:26:53.146068 17736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea775984f7d4755aead01d5c41d8d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:53 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:38722
I20260928 12:26:53.146054 17602 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea775984f7d4755aead01d5c41d8d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:53 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:54138
I20260928 12:26:53.146270 17974 tablet_replica.cc:330] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9: stopping tablet replica
I20260928 12:26:53.146260 17870 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea775984f7d4755aead01d5c41d8d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:26:53 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:42196
I20260928 12:26:53.146363 17974 raft_consensus.cc:2229] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:26:53.146467 17974 raft_consensus.cc:2258] T 1ea775984f7d4755aead01d5c41d8d59 P 176e6ec0dd4d4f60a1d9b68615ace0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:53.146621 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17539
I20260928 12:26:53.148239 17975 tablet_replica.cc:330] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: stopping tablet replica
I20260928 12:26:53.148326 17975 raft_consensus.cc:2229] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:26:53.148447 17975 raft_consensus.cc:2258] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:26:53.148883 17975 ts_tablet_manager.cc:1905] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:26:53.150859 17975 ts_tablet_manager.cc:1918] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:26:53.150933 17975 log.cc:1196] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/1ea775984f7d4755aead01d5c41d8d59
I20260928 12:26:53.151181 17975 ts_tablet_manager.cc:1939] T 1ea775984f7d4755aead01d5c41d8d59 P 29ee768ab1a34dab838eb25f2ec7464d: Deleting consensus metadata
I20260928 12:26:53.151568 17461 catalog_manager.cc:4841] TS 29ee768ab1a34dab838eb25f2ec7464d (127.7.19.67:43423): tablet 1ea775984f7d4755aead01d5c41d8d59 (table test-table [id=c72bdb800f0046ab96a2294e1bbcba6c]) successfully deleted
W20260928 12:26:53.152428 17462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:38513 (error 108)
I20260928 12:26:53.152701 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17673
I20260928 12:26:53.160778 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17807
I20260928 12:26:53.162366 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.162220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:38513 (local address 127.0.0.1:54154)
0928 12:26:53.162285 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:53.162331 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 12:26:53.162421 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:26:53.162294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:32889 (local address 127.0.0.1:38732)
0928 12:26:53.162332 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:26:53.162355 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.66:32889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 12:26:53.162523 17462 catalog_manager.cc:4584] TS 176e6ec0dd4d4f60a1d9b68615ace0e9 (127.7.19.65:38513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ea775984f7d4755aead01d5c41d8d59: Network error: Client connection negotiation failed: client connection to 127.7.19.65:38513: connect: Connection refused (error 111)
I20260928 12:26:53.165256 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 17445
2026-09-28T12:26:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1996 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:26:53 dist-test-slave-cb7x krb5kdc[17979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:26:53 dist-test-slave-cb7x krb5kdc[17979](info): set up 2 sockets
Sep 28 12:26:53 dist-test-slave-cb7x krb5kdc[17979](info): commencing operation
krb5kdc: starting...
W20260928 12:26:54.217695 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:26:54 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:26:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/rangeradmin_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/rangeradmin_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/rangerlookup_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/rangerlookup_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
I20260928 12:26:54.388115 7245 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMIVMOs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/postgres -l logfile start
2026-09-28 12:26:57.013 UTC [18009] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:26:57.013 UTC [18009] LOG: listening on IPv4 address "127.7.19.84", port 32963
2026-09-28 12:26:57.017 UTC [18009] LOG: listening on Unix socket "/tmp/.s.PGSQL.32963"
2026-09-28 12:26:57.054 UTC [18012] LOG: database system was shut down at 2026-09-28 12:26:55 UTC
2026-09-28 12:26:57.058 UTC [18009] LOG: database system is ready to accept connections
I20260928 12:26:58.033910 7245 mini_postgres.cc:96] Postgres bound to 32963
2026-09-28 12:26:58.040 UTC [18020] FATAL: database "slave" does not exist
127.7.19.84:32963 - accepting connections
I20260928 12:26:58.041179 7245 mini_ranger.cc:161] Starting Ranger...
I20260928 12:26:58.058928 7245 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:26:58.592428 7245 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:26:58.592515 7245 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-admin
2026-09-28 12:26:58,856 [I] DB FLAVOR :POSTGRES
2026-09-28 12:26:58,856 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:26:58,856 [I] Checking connection..
2026-09-28 12:26:58,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:26:59,094 [I] Checking connection passed.
2026-09-28 12:26:59,094 [I] --------- Verifying version history table ---------
2026-09-28 12:26:59,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:26:59,359 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:26:59,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:26:59,605 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:26:59,605 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:26:59,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:26:59.855 UTC [18084] WARNING: there is no transaction in progress
2026-09-28 12:26:59,861 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:26:59,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:27:00,117 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:27:00,118 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:27:00,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:27:00,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:27:00,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cb7x.c.gcp-upstream.internal','N') ;"
2026-09-28 12:27:00,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:27:01,080 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:27:01,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:27:01,326 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:27:01,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:27:01,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:27:01,810 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:27:01,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:27:02.759 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.776 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.793 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.809 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.825 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.843 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.896 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.905 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.921 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.931 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.942 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.955 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.964 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.970 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.977 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:02.983 UTC [18192] WARNING: there is no transaction in progress
2026-09-28 12:27:03,659 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:27:03,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:27:03,906 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:27:03,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:27:04,172 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:27:04,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:27:04,435 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:27:04,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:27:04,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:27:04,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cb7x.c.gcp-upstream.internal';"
2026-09-28 12:27:05,141 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:27:05,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:27:05,385 [I] DB_PATCHES have already been applied
I20260928 12:27:05.391059 7245 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMIVMOs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:27:05.391121 7245 mini_ranger.cc:193] Using host: 127.7.19.84
I20260928 12:27:05.394126 7245 mini_ranger.cc:227] Ranger admin URL: http://127.7.19.84:53527
Sep 28, 2026 12:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:27:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:27:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:27:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:27:06.521450 7245 mini_ranger.cc:160] Time spent starting Ranger: real 8.480s user 0.000s sys 0.009s
Sep 28, 2026 12:27:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:27:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 12:27:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:27:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:27:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:27:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:27:27.379952 7245 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/rangerkms_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/rangerkms_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.19.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.19.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:27:27 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598447, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:27:27.465263 7245 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:27:27.474663 7245 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:27:28.394606 7245 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:27:28.394692 7245 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms
I20260928 12:27:28.394716 7245 mini_ranger_kms.cc:227] Using postgres at 127.7.19.84:32963
2026-09-28 12:27:28,583 [I] DB FLAVOR :POSTGRES
2026-09-28 12:27:28,583 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:27:28,583 [I] Checking connection
2026-09-28 12:27:28,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:27:28,817 [I] connection success
2026-09-28 12:27:28,817 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:27:28,817 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:27:28,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:27:29,072 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:27:29,072 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:27:29,072 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:27:29,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:32963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:27:29,326 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:27:29.821130 7245 mini_ranger_kms.cc:310] Created kms service
I20260928 12:27:30.031273 7245 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:27:30.074707 7245 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:27:30 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598447, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:27:30.483608 7245 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:27:30.483731 7245 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:27:30.483753 7245 mini_ranger_kms.cc:240] Using host: 127.7.19.84
I20260928 12:27:30.487479 7245 mini_ranger_kms.cc:276] Ranger KMS PID: 18372
I20260928 12:27:30.487614 7245 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.19.84:53985
Sep 28, 2026 12:27:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:27:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:27:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:27:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:27:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:27:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 12:27:32 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598452, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.19.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:27:32.763345 7245 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.298s user 0.004s sys 0.164s
I20260928 12:27:32.763449 7245 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:27:32.763523 7245 mini_ranger_kms.cc:401] 127.7.19.84:53985/kms/v1/keys
Sep 28 12:27:33 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598452, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.19.84@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
Sep 28 12:27:33 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598452, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.19.84@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
Sep 28 12:27:35 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:27:35.652877 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43715
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:43715
--ranger_config_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:53985/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:27:35.734824 18423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:27:35.735031 18423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:27:35.735052 18423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:27:35.736121 18423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:27:35.736156 18423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:27:35.736171 18423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:27:35.736183 18423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:27:35.736195 18423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:27:35.737788 18423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.19.84:53985/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:43715
--ranger_config_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:27:35.737979 18423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:27:35.738155 18423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:27:35.743016 18423 server_base.cc:1034] running on GCE node
W20260928 12:27:35.742988 18431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:35.742988 18429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:35.743032 18428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:27:35.743463 18423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:27:35.743741 18423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:27:35.744899 18423 hybrid_clock.cc:648] HybridClock initialized: now 1790598455744872 us; error 35 us; skew 500 ppm
Sep 28 12:27:35 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598455, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:27:35.759155 18423 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:27:35.759639 18423 webserver.cc:460] Webserver started at http://127.7.19.126:44971/ using document root <none> and password file <none>
I20260928 12:27:35.759876 18423 fs_manager.cc:359] Metadata directory not provided
I20260928 12:27:35.759936 18423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:27:35.760075 18423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:27:35 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598455, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:27:35.815037 18423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "4dcb4cc603014f94b50cbd612d44ae21"
format_stamp: "Formatted at 2026-09-28 12:27:35 on dist-test-slave-cb7x"
server_key: "801da88899dd3d1ccc68fc5da951bcd2"
server_key_iv: "14eff60790523d3ed6608fedc216d18d"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:35.815493 18423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "4dcb4cc603014f94b50cbd612d44ae21"
format_stamp: "Formatted at 2026-09-28 12:27:35 on dist-test-slave-cb7x"
server_key: "801da88899dd3d1ccc68fc5da951bcd2"
server_key_iv: "14eff60790523d3ed6608fedc216d18d"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:35.817299 18423 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 12:27:35.831190 18438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:35.831418 18423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:27:35.831511 18423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "4dcb4cc603014f94b50cbd612d44ae21"
format_stamp: "Formatted at 2026-09-28 12:27:35 on dist-test-slave-cb7x"
server_key: "801da88899dd3d1ccc68fc5da951bcd2"
server_key_iv: "14eff60790523d3ed6608fedc216d18d"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:35.831620 18423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:27:35.846529 18423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:27:35.848248 18423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:27:35.848392 18423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:27:35.853325 18423 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:43715
I20260928 12:27:35.853408 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:43715 every 8 connection(s)
I20260928 12:27:35.853713 18423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:27:35.854359 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:27:35.856599 18491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21: Bootstrap starting.
I20260928 12:27:35.857234 18491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21: Neither blocks nor log segments found. Creating new log.
I20260928 12:27:35.857481 18491 log.cc:824] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21: Log is configured to *not* fsync() on all Append() calls
I20260928 12:27:35.858131 18491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21: No bootstrap required, opened a new log
I20260928 12:27:35.858722 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 18423
I20260928 12:27:35.858831 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:27:35.859006 7245 external_mini_cluster.cc:1428] Setting key aa3782a2b3f71736e642d677837b96f8
I20260928 12:27:35.861078 18491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcb4cc603014f94b50cbd612d44ae21" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43715 } }
I20260928 12:27:35.861243 18491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:27:35.861305 18491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dcb4cc603014f94b50cbd612d44ae21, State: Initialized, Role: FOLLOWER
I20260928 12:27:35.861471 18491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcb4cc603014f94b50cbd612d44ae21" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43715 } }
I20260928 12:27:35.861567 18491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:27:35.861611 18491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:27:35.861661 18491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:27:35.862316 18491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcb4cc603014f94b50cbd612d44ae21" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43715 } }
I20260928 12:27:35.862437 18491 leader_election.cc:302] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dcb4cc603014f94b50cbd612d44ae21; no voters:
I20260928 12:27:35.862608 18491 leader_election.cc:288] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:27:35.862665 18496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:27:35.862874 18491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:27:35.862893 18496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [term 1 LEADER]: Becoming Leader. State: Replica: 4dcb4cc603014f94b50cbd612d44ae21, State: Running, Role: LEADER
I20260928 12:27:35.863126 18496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcb4cc603014f94b50cbd612d44ae21" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43715 } }
I20260928 12:27:35.863507 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dcb4cc603014f94b50cbd612d44ae21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcb4cc603014f94b50cbd612d44ae21" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43715 } } }
I20260928 12:27:35.863590 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [sys.catalog]: This master's current role is: LEADER
I20260928 12:27:35.863724 18498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dcb4cc603014f94b50cbd612d44ae21. Latest consensus state: current_term: 1 leader_uuid: "4dcb4cc603014f94b50cbd612d44ae21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcb4cc603014f94b50cbd612d44ae21" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43715 } } }
I20260928 12:27:35.863772 18498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21 [sys.catalog]: This master's current role is: LEADER
I20260928 12:27:35.864203 18505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:27:35.864442 18491 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:27:35.864748 18505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:27:35.869951 18505 catalog_manager.cc:1347] Generated new cluster ID: c8d2d033fc39474cb1e43b6c84381654
I20260928 12:27:35.870009 18505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:27:35 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:27:35.876237 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:35.859531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.7.19.126:43715)
0928 12:27:35.859686 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:35.859694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:35.859719 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:27:35.861087 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:35.861095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:35.861158 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:35.861337 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:27:35.862184 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:35.863754 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:35.866115 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:35.867796 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:35.868009 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:35.871268 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:27:35.871271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:27:35.871272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:27:35.871283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:27:35.873519 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:35.873926 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:35.873930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:35.873930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:35.874652 (+ 722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:35.874885 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:35.874886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:35.874886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:35.875228 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:27:35.875389 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:35.875865 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 12:27:35.876003 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260928 12:27:35.884498 18505 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:27:35.885236 18505 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:27:35.896317 18505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21: Generated new TSK 0
I20260928 12:27:35.896539 18505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:27:36.553969 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:53985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43715
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:27:36.666363 18536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:27:36.666584 18536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:27:36.666608 18536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:27:36.667773 18536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:27:36.667819 18536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:27:36.667845 18536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:27:36.669709 18536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.19.84:53985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:27:36.669965 18536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:27:36.670261 18536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:27:36.676662 18544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:36.676662 18541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:36.676961 18542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:27:36.676961 18536 server_base.cc:1034] running on GCE node
I20260928 12:27:36.677222 18536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:27:36.677495 18536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:27:36.678668 18536 hybrid_clock.cc:648] HybridClock initialized: now 1790598456678653 us; error 32 us; skew 500 ppm
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:27:36.681315 18536 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:27:36.681815 18536 webserver.cc:460] Webserver started at http://127.7.19.65:44581/ using document root <none> and password file <none>
I20260928 12:27:36.682062 18536 fs_manager.cc:359] Metadata directory not provided
I20260928 12:27:36.682128 18536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:27:36.682261 18536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:27:36.695865 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "b038a4d3496e456089e27feceb544c10"
format_stamp: "Formatted at 2026-09-28 12:27:36 on dist-test-slave-cb7x"
server_key: "cffd1972fd72fa75101e29cfbffd7b43"
server_key_iv: "44a038f40ff217ccdf6dfac7ae3edffb"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:36.696410 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "b038a4d3496e456089e27feceb544c10"
format_stamp: "Formatted at 2026-09-28 12:27:36 on dist-test-slave-cb7x"
server_key: "cffd1972fd72fa75101e29cfbffd7b43"
server_key_iv: "44a038f40ff217ccdf6dfac7ae3edffb"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:36.698822 18536 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:27:36.712320 18551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:36.712563 18536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:27:36.712733 18536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "b038a4d3496e456089e27feceb544c10"
format_stamp: "Formatted at 2026-09-28 12:27:36 on dist-test-slave-cb7x"
server_key: "cffd1972fd72fa75101e29cfbffd7b43"
server_key_iv: "44a038f40ff217ccdf6dfac7ae3edffb"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:36.712878 18536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:27:36.738014 18536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:27:36.740492 18536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:27:36.740721 18536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:27:36.741042 18536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:27:36.741449 18536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:27:36.741530 18536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:36.741590 18536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:27:36.741675 18536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:36.747741 18536 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:34147
I20260928 12:27:36.748221 18536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:27:36.750630 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 18536
I20260928 12:27:36.750766 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:27:36.750993 7245 external_mini_cluster.cc:1428] Setting key e5d73358d758d05f3a3403e595d75169
I20260928 12:27:36.750448 18666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:34147 every 8 connection(s)
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
I20260928 12:27:36.763404 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:36.749032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:54671 (local address 127.7.19.126:43715)
0928 12:27:36.749179 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:36.749182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:36.751857 (+ 2675us) server_negotiation.cc:407] Connection header received
0928 12:27:36.751880 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:36.751881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:36.751894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:36.751967 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:27:36.752568 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.753142 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.754603 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.754936 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.755006 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:36.759487 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:27:36.759489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:27:36.759490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:27:36.759499 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:27:36.761308 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:36.761821 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:36.761823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:36.761823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:36.761911 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:36.762250 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:36.762252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:36.762252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:36.762409 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:27:36.762458 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:36.763157 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 12:27:36.763272 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
I20260928 12:27:36.764097 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:36.748982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43715 (local address 127.7.19.65:54671)
0928 12:27:36.750639 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:36.750673 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:36.751723 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:36.752105 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:36.752109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:36.752285 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:27:36.752474 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:36.752478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.753233 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.753234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:36.754509 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:36.754512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.756762 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.756763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:36.756927 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:36.757595 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:27:36.757605 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:27:36.759347 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:27:36.761394 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:27:36.761401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:27:36.761403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:27:36.761728 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:27:36.761989 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:27:36.761990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:27:36.761990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:27:36.762151 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:27:36.762514 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:27:36.762520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:27:36.762865 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 12:27:36.763520 (+ 655us) client_negotiation.cc:240] Negotiation successful
0928 12:27:36.763756 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1624,"thread_start_us":85,"threads_started":1}
I20260928 12:27:36.765975 18667 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43715
I20260928 12:27:36.766076 18667 heartbeater.cc:464] Registering TS with master...
I20260928 12:27:36.766269 18667 heartbeater.cc:511] Master 127.7.19.126:43715 requested a full tablet report, sending...
I20260928 12:27:36.766866 18455 ts_manager.cc:194] Registered new tserver with Master: b038a4d3496e456089e27feceb544c10 (127.7.19.65:34147)
I20260928 12:27:36.767822 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:54671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:27:36.789489 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:53985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43715
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:27:36.905267 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:27:36.905503 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:27:36.905543 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:27:36.906810 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:27:36.906875 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:27:36.906922 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:27:36.908950 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.19.84:53985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:27:36.909214 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:27:36.909705 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 12:27:36.915409 18683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:36.915462 18680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:36.915781 18681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:27:36.915792 18673 server_base.cc:1034] running on GCE node
I20260928 12:27:36.916026 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:27:36.916308 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:27:36.917451 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790598456917423 us; error 39 us; skew 500 ppm
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:27:36.920815 18673 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:27:36.921417 18673 webserver.cc:460] Webserver started at http://127.7.19.66:39503/ using document root <none> and password file <none>
I20260928 12:27:36.921641 18673 fs_manager.cc:359] Metadata directory not provided
I20260928 12:27:36.921706 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:27:36.921842 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:27:36.944653 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "0c78f62d9eef4dc5b9648f8514727436"
format_stamp: "Formatted at 2026-09-28 12:27:36 on dist-test-slave-cb7x"
server_key: "f4085dd25386c6d18621134a7bd368d5"
server_key_iv: "3f05f8947670057d9df9b95ceebb45c2"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:36.945690 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "0c78f62d9eef4dc5b9648f8514727436"
format_stamp: "Formatted at 2026-09-28 12:27:36 on dist-test-slave-cb7x"
server_key: "f4085dd25386c6d18621134a7bd368d5"
server_key_iv: "3f05f8947670057d9df9b95ceebb45c2"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:36.950489 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 12:27:36.969060 18690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:36.969516 18673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 12:27:36.969637 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "0c78f62d9eef4dc5b9648f8514727436"
format_stamp: "Formatted at 2026-09-28 12:27:36 on dist-test-slave-cb7x"
server_key: "f4085dd25386c6d18621134a7bd368d5"
server_key_iv: "3f05f8947670057d9df9b95ceebb45c2"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:36.969748 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:27:36.982981 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:27:36.985750 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:27:36.985908 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:27:36.986145 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:27:36.986555 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:27:36.986640 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:36.986711 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:27:36.986769 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:36.993920 18673 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:38781
I20260928 12:27:36.994045 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:38781 every 8 connection(s)
I20260928 12:27:36.994395 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:27:36.997439 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 18673
I20260928 12:27:36.997530 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:27:36.997723 7245 external_mini_cluster.cc:1428] Setting key de2277f879acecfbac0b396051f942ff
Sep 28 12:27:36 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:27:37.006381 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:36.995145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:41327 (local address 127.7.19.126:43715)
0928 12:27:36.995198 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:36.995200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:36.995459 (+ 259us) server_negotiation.cc:407] Connection header received
0928 12:27:36.996307 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:36.996309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:36.996322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:36.996355 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:27:36.997093 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.997896 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.999400 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.999726 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.999770 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.002526 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:27:37.002527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:27:37.002528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:27:37.002535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:27:37.004709 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:37.005232 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:37.005233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:37.005234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:37.005291 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:37.005606 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:37.005606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:37.005607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:37.005743 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:27:37.005785 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.006170 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.006262 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:27:37.007135 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:36.995141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43715 (local address 127.7.19.66:41327)
0928 12:27:36.995311 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:36.995362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:36.996096 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:36.996528 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:36.996532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:36.996792 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:27:36.996995 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:36.997000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.998009 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.998011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:36.999291 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:36.999293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:36.999868 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:36.999869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.000026 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.000684 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:27:37.000695 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:27:37.002411 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:27:37.004827 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:27:37.004835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:27:37.004838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:27:37.005134 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:27:37.005375 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:27:37.005376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:27:37.005376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:27:37.005529 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:27:37.005832 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:27:37.005837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:27:37.006017 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.006738 (+ 721us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.006905 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:27:37.007781 18804 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43715
I20260928 12:27:37.007876 18804 heartbeater.cc:464] Registering TS with master...
I20260928 12:27:37.008064 18804 heartbeater.cc:511] Master 127.7.19.126:43715 requested a full tablet report, sending...
I20260928 12:27:37.008504 18455 ts_manager.cc:194] Registered new tserver with Master: 0c78f62d9eef4dc5b9648f8514727436 (127.7.19.66:38781)
I20260928 12:27:37.009040 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:41327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:27:37.027566 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:53985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43715
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:27:37.150230 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:27:37.150570 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:27:37.150660 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:27:37.152379 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:27:37.152493 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:27:37.152578 18810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:27:37.154778 18810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.19.84:53985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:27:37.155237 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:27:37.155557 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:27:37.162688 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:37.162847 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:27:37.163108 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:27:37.164341 18810 server_base.cc:1034] running on GCE node
I20260928 12:27:37.164587 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:27:37.164897 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:27:37.166076 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790598457166055 us; error 43 us; skew 500 ppm
Sep 28 12:27:37 dist-test-slave-cb7x krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598457, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:27:37.169247 18810 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:27:37.169770 18810 webserver.cc:460] Webserver started at http://127.7.19.67:39879/ using document root <none> and password file <none>
I20260928 12:27:37.170014 18810 fs_manager.cc:359] Metadata directory not provided
I20260928 12:27:37.170133 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:27:37.170270 18810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:27:37 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598457, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:27:37.184319 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "d7a01ae154b3424184d1299add9e86e5"
format_stamp: "Formatted at 2026-09-28 12:27:37 on dist-test-slave-cb7x"
server_key: "f4c7318132f1471bfd61a75a47df8b85"
server_key_iv: "52a0bc2ebd2aeffef756d4462e1eb459"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:37.184820 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "d7a01ae154b3424184d1299add9e86e5"
format_stamp: "Formatted at 2026-09-28 12:27:37 on dist-test-slave-cb7x"
server_key: "f4c7318132f1471bfd61a75a47df8b85"
server_key_iv: "52a0bc2ebd2aeffef756d4462e1eb459"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:37.186395 18810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:27:37.201049 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:37.201231 18810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:27:37.201318 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "d7a01ae154b3424184d1299add9e86e5"
format_stamp: "Formatted at 2026-09-28 12:27:37 on dist-test-slave-cb7x"
server_key: "f4c7318132f1471bfd61a75a47df8b85"
server_key_iv: "52a0bc2ebd2aeffef756d4462e1eb459"
server_key_version: "kuduclusterkey@0"
I20260928 12:27:37.201416 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:27:37.215654 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:27:37.218098 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:27:37.218303 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:27:37.218588 18810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:27:37.219019 18810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:27:37.219107 18810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:37.219182 18810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:27:37.219239 18810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:27:37.226099 18810 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:40581
I20260928 12:27:37.226625 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:27:37.228159 18938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:40581 every 8 connection(s)
I20260928 12:27:37.235625 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 18810
I20260928 12:27:37.235726 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
Sep 28 12:27:37 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598457, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:27:37.236313 7245 external_mini_cluster.cc:1428] Setting key deed1bab18db6d31d74b8d706df5a1af
I20260928 12:27:37.240922 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.229177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:36465 (local address 127.7.19.126:43715)
0928 12:27:37.229222 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.229224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.229552 (+ 328us) server_negotiation.cc:407] Connection header received
0928 12:27:37.230233 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.230235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.230247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.230314 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:27:37.231109 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.231905 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.233497 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.233739 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.233808 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.237504 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:27:37.237505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:27:37.237506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:27:37.237513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:27:37.239292 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:37.239772 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:37.239773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:37.239774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:37.239835 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:37.240140 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:37.240141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:37.240142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:37.240293 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:27:37.240366 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.240737 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.240826 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:27:37.241128 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.228859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43715 (local address 127.7.19.67:36465)
0928 12:27:37.229387 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.229435 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:37.230096 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:37.230462 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:37.230467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:37.230717 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:27:37.230995 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.231001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.232059 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.232060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.233385 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.233388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.233875 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.233876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.234048 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.234597 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:27:37.234605 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:27:37.237404 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:27:37.239382 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:27:37.239389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:27:37.239392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:27:37.239689 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:27:37.239910 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:27:37.239911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:27:37.239911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:27:37.240063 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:27:37.240366 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:27:37.240372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:27:37.240559 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.240706 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.240895 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":80,"threads_started":1}
I20260928 12:27:37.241788 18939 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43715
I20260928 12:27:37.241897 18939 heartbeater.cc:464] Registering TS with master...
I20260928 12:27:37.242092 18939 heartbeater.cc:511] Master 127.7.19.126:43715 requested a full tablet report, sending...
I20260928 12:27:37.242468 18455 ts_manager.cc:194] Registered new tserver with Master: d7a01ae154b3424184d1299add9e86e5 (127.7.19.67:40581)
I20260928 12:27:37.243012 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:36465
I20260928 12:27:37.247952 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:27:37.257983 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.249743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.7.19.126:43715)
0928 12:27:37.249784 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.249786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.250069 (+ 283us) server_negotiation.cc:407] Connection header received
0928 12:27:37.250165 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.250166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.250177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.250206 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:27:37.250983 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.251739 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.252887 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.253146 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.253194 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.254798 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:27:37.254800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:27:37.254800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:27:37.254807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:27:37.256565 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:37.257014 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:37.257015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:37.257016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:37.257080 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:27:37.257304 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:27:37.257305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:27:37.257306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:27:37.257460 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:27:37.257564 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.257791 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.257885 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:27:37.259721 18455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:27:37.261518 18455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:27:37.270538 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.264553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40581 (local address 127.0.0.1:43112)
0928 12:27:37.265171 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.265219 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:37.265283 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:37.265691 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:37.265695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:37.265697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:27:37.265772 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.265774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.267145 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.267146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.268715 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.268719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.270197 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.270198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.270291 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.270306 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.270382 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.270453 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":537,"threads_started":1}
I20260928 12:27:37.273125 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.264827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:38781 (local address 127.0.0.1:46332)
0928 12:27:37.268900 (+ 4073us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.268911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:37.268969 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:37.269235 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:37.269236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:37.269237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:27:37.269279 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.269281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.270532 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.270533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.271661 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.271664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.272833 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.272834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.272909 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.272913 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.272967 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.273007 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4052,"thread_start_us":72,"threads_started":1}
I20260928 12:27:37.273371 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.265393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.7.19.66:38781)
0928 12:27:37.268830 (+ 3437us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.268835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.269027 (+ 192us) server_negotiation.cc:407] Connection header received
0928 12:27:37.269069 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.269072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.269088 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.269156 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:27:37.269354 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.270433 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.271780 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.272586 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.272655 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.272761 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.273251 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.273294 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3396,"thread_start_us":100,"threads_started":1}
I20260928 12:27:37.274196 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9e0d820ac640c689fe138c64cb51c5 (DEFAULT_TABLE table=test-table [id=92bfa6554caf462b8a375d202eac75c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:27:37.274602 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9e0d820ac640c689fe138c64cb51c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:27:37.274739 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.264954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.7.19.67:40581)
0928 12:27:37.265446 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.265452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.265464 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:27:37.265506 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.265508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.265525 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.265601 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:27:37.265936 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.267036 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.268837 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.270002 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.270077 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.270159 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.271461 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.271506 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":73,"threads_started":1}
I20260928 12:27:37.275372 18873 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9e0d820ac640c689fe138c64cb51c5 (DEFAULT_TABLE table=test-table [id=92bfa6554caf462b8a375d202eac75c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:27:37.275717 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9e0d820ac640c689fe138c64cb51c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:27:37.278496 18957 tablet_bootstrap.cc:492] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Bootstrap starting.
I20260928 12:27:37.279695 18957 tablet_bootstrap.cc:654] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Neither blocks nor log segments found. Creating new log.
I20260928 12:27:37.280146 18957 log.cc:824] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Log is configured to *not* fsync() on all Append() calls
I20260928 12:27:37.280583 18958 tablet_bootstrap.cc:492] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Bootstrap starting.
I20260928 12:27:37.281862 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.265080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:34147 (local address 127.0.0.1:46592)
0928 12:27:37.270984 (+ 5904us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.270995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:37.271078 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:37.271336 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:37.271336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:37.271338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:27:37.271369 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.271370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.273118 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.273119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.279947 (+ 6828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.279952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.281562 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.281563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.281651 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.281656 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.281714 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.281778 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5893,"thread_start_us":62,"threads_started":1}
I20260928 12:27:37.282179 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.265584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.7.19.65:34147)
0928 12:27:37.265699 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.265704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.271047 (+ 5343us) server_negotiation.cc:407] Connection header received
0928 12:27:37.271164 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.271167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.271185 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.271242 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:27:37.271902 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.273013 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.280104 (+ 7091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.280962 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.281407 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.281488 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.282021 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.282112 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
I20260928 12:27:37.282688 18958 tablet_bootstrap.cc:654] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Neither blocks nor log segments found. Creating new log.
I20260928 12:27:37.283048 18958 log.cc:824] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:27:37.283017 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9e0d820ac640c689fe138c64cb51c5 (DEFAULT_TABLE table=test-table [id=92bfa6554caf462b8a375d202eac75c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:27:37.283344 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9e0d820ac640c689fe138c64cb51c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:27:37.283877 18957 tablet_bootstrap.cc:492] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: No bootstrap required, opened a new log
I20260928 12:27:37.283963 18957 ts_tablet_manager.cc:1397] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 12:27:37.286114 18957 raft_consensus.cc:348] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.286402 18957 raft_consensus.cc:374] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:27:37.286494 18957 raft_consensus.cc:729] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c78f62d9eef4dc5b9648f8514727436, State: Initialized, Role: FOLLOWER
I20260928 12:27:37.286696 18957 consensus_queue.cc:260] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.287034 18961 tablet_bootstrap.cc:492] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: Bootstrap starting.
I20260928 12:27:37.287072 18957 ts_tablet_manager.cc:1428] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:27:37.287189 18804 heartbeater.cc:503] Master 127.7.19.126:43715 was elected leader, sending a full tablet report...
I20260928 12:27:37.287760 18961 tablet_bootstrap.cc:654] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: Neither blocks nor log segments found. Creating new log.
I20260928 12:27:37.288086 18961 log.cc:824] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: Log is configured to *not* fsync() on all Append() calls
I20260928 12:27:37.288829 18958 tablet_bootstrap.cc:492] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: No bootstrap required, opened a new log
I20260928 12:27:37.288905 18961 tablet_bootstrap.cc:492] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: No bootstrap required, opened a new log
I20260928 12:27:37.288971 18961 ts_tablet_manager.cc:1397] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:27:37.288975 18958 ts_tablet_manager.cc:1397] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260928 12:27:37.291287 18958 raft_consensus.cc:348] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.291446 18958 raft_consensus.cc:374] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:27:37.291479 18958 raft_consensus.cc:729] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a01ae154b3424184d1299add9e86e5, State: Initialized, Role: FOLLOWER
I20260928 12:27:37.291461 18961 raft_consensus.cc:348] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.291611 18961 raft_consensus.cc:374] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:27:37.291642 18961 raft_consensus.cc:729] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b038a4d3496e456089e27feceb544c10, State: Initialized, Role: FOLLOWER
I20260928 12:27:37.291620 18958 consensus_queue.cc:260] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.291887 18958 ts_tablet_manager.cc:1428] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:27:37.292008 18939 heartbeater.cc:503] Master 127.7.19.126:43715 was elected leader, sending a full tablet report...
I20260928 12:27:37.293254 18961 consensus_queue.cc:260] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.293570 18961 ts_tablet_manager.cc:1428] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 12:27:37.293594 18667 heartbeater.cc:503] Master 127.7.19.126:43715 was elected leader, sending a full tablet report...
Sep 28 12:27:37 dist-test-slave-cb7x krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598456, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:27:37.473374 18964 raft_consensus.cc:482] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:27:37.473546 18964 raft_consensus.cc:504] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.473942 18964 leader_election.cc:288] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c78f62d9eef4dc5b9648f8514727436 (127.7.19.66:38781), b038a4d3496e456089e27feceb544c10 (127.7.19.65:34147)
I20260928 12:27:37.477989 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.474399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:34485 (local address 127.7.19.65:34147)
0928 12:27:37.474442 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.474444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.474519 (+ 75us) server_negotiation.cc:407] Connection header received
0928 12:27:37.474549 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.474550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.474561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.474611 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:27:37.474806 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.475525 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.477014 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.477619 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.477664 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.477673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.477878 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.477917 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:27:37.478228 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.474216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:34147 (local address 127.7.19.67:34485)
0928 12:27:37.474451 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.474461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:37.474509 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:37.474654 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:37.474656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:37.474658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:27:37.474734 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.474735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.475620 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.475621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.476915 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.476919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.477730 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.477731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.477802 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.477806 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.478105 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.478144 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260928 12:27:37.478619 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9e0d820ac640c689fe138c64cb51c5" candidate_uuid: "d7a01ae154b3424184d1299add9e86e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b038a4d3496e456089e27feceb544c10" is_pre_election: true
I20260928 12:27:37.478777 18621 raft_consensus.cc:2454] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7a01ae154b3424184d1299add9e86e5 in term 0.
I20260928 12:27:37.478986 18829 leader_election.cc:302] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b038a4d3496e456089e27feceb544c10, d7a01ae154b3424184d1299add9e86e5; no voters:
I20260928 12:27:37.479142 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.474367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:53037 (local address 127.7.19.66:38781)
0928 12:27:37.474420 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.474422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.474434 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:27:37.474464 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.474465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.474476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.474539 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:27:37.474732 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.475498 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.477347 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.477976 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.478047 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.478058 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.479018 (+ 960us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.479067 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:27:37.479173 18964 raft_consensus.cc:2781] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:27:37.479214 18964 raft_consensus.cc:482] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:27:37.479234 18964 raft_consensus.cc:3037] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 0 FOLLOWER]: Advancing to term 1
W20260928 12:27:37.479506 18940 tablet.cc:2369] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:27:37.479597 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.474126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:38781 (local address 127.7.19.67:53037)
0928 12:27:37.474202 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.474214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:27:37.474281 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:27:37.474585 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:27:37.474586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:27:37.474587 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:27:37.474657 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.474659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.475619 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.475620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.477226 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:27:37.477229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.478745 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.478746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:27:37.478831 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.478835 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:27:37.478891 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:27:37.478937 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 12:27:37.480005 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9e0d820ac640c689fe138c64cb51c5" candidate_uuid: "d7a01ae154b3424184d1299add9e86e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c78f62d9eef4dc5b9648f8514727436" is_pre_election: true
I20260928 12:27:37.480155 18758 raft_consensus.cc:2454] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7a01ae154b3424184d1299add9e86e5 in term 0.
I20260928 12:27:37.480140 18964 raft_consensus.cc:504] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.480260 18964 leader_election.cc:288] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [CANDIDATE]: Term 1 election: Requested vote from peers 0c78f62d9eef4dc5b9648f8514727436 (127.7.19.66:38781), b038a4d3496e456089e27feceb544c10 (127.7.19.65:34147)
I20260928 12:27:37.480545 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9e0d820ac640c689fe138c64cb51c5" candidate_uuid: "d7a01ae154b3424184d1299add9e86e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b038a4d3496e456089e27feceb544c10"
I20260928 12:27:37.480567 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9e0d820ac640c689fe138c64cb51c5" candidate_uuid: "d7a01ae154b3424184d1299add9e86e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c78f62d9eef4dc5b9648f8514727436"
I20260928 12:27:37.480618 18621 raft_consensus.cc:3037] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:27:37.480646 18758 raft_consensus.cc:3037] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:27:37.481462 18621 raft_consensus.cc:2454] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7a01ae154b3424184d1299add9e86e5 in term 1.
I20260928 12:27:37.481734 18829 leader_election.cc:302] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b038a4d3496e456089e27feceb544c10, d7a01ae154b3424184d1299add9e86e5; no voters:
I20260928 12:27:37.481850 18964 raft_consensus.cc:2781] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:27:37.482002 18964 raft_consensus.cc:686] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 1 LEADER]: Becoming Leader. State: Replica: d7a01ae154b3424184d1299add9e86e5, State: Running, Role: LEADER
I20260928 12:27:37.482086 18758 raft_consensus.cc:2454] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7a01ae154b3424184d1299add9e86e5 in term 1.
I20260928 12:27:37.482086 18964 consensus_queue.cc:237] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } }
I20260928 12:27:37.482903 18454 catalog_manager.cc:5495] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7a01ae154b3424184d1299add9e86e5 (127.7.19.67). New cstate: current_term: 1 leader_uuid: "d7a01ae154b3424184d1299add9e86e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a01ae154b3424184d1299add9e86e5" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 } health_report { overall_health: UNKNOWN } } }
W20260928 12:27:37.495342 18805 tablet.cc:2369] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:27:37.498123 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.495662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.7.19.67:40581)
0928 12:27:37.495798 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 12:27:37.495800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:27:37.495805 (+ 5us) server_negotiation.cc:407] Connection header received
0928 12:27:37.495838 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:27:37.495838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:27:37.495856 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:27:37.495930 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:27:37.496041 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.496514 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.497369 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:27:37.497581 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:27:37.497655 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:27:37.497792 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:27:37.497900 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:27:37.497993 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 12:27:37.498013 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 12:27:37.498033 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 12:27:37.499867 18621 raft_consensus.cc:1261] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10 [term 1 FOLLOWER]: Refusing update from remote peer d7a01ae154b3424184d1299add9e86e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:27:37.500216 18964 consensus_queue.cc:1035] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b038a4d3496e456089e27feceb544c10" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:27:37.500523 18758 raft_consensus.cc:1261] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 1 FOLLOWER]: Refusing update from remote peer d7a01ae154b3424184d1299add9e86e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:27:37.500829 18964 consensus_queue.cc:1035] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c78f62d9eef4dc5b9648f8514727436" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:27:37.501062 18668 tablet.cc:2369] T 3f9e0d820ac640c689fe138c64cb51c5 P b038a4d3496e456089e27feceb544c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:27:37.504051 18970 mvcc.cc:205] Tried to move back new op lower bound from 7334291281916735488 to 7334291281847058432. Current Snapshot: MvccSnapshot[applied={T|T < 7334291281916735488}]
I20260928 12:27:37.504136 18972 mvcc.cc:205] Tried to move back new op lower bound from 7334291281916735488 to 7334291281847058432. Current Snapshot: MvccSnapshot[applied={T|T < 7334291281916735488}]
I20260928 12:27:37.506147 18971 mvcc.cc:205] Tried to move back new op lower bound from 7334291281916735488 to 7334291281847058432. Current Snapshot: MvccSnapshot[applied={T|T < 7334291281916735488}]
I20260928 12:27:37.507287 18454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:27:37.507385 18454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:27:37.508361 18454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dcb4cc603014f94b50cbd612d44ae21: Sending DeleteTablet for 3 replicas of tablet 3f9e0d820ac640c689fe138c64cb51c5
I20260928 12:27:37.508941 18873 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9e0d820ac640c689fe138c64cb51c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:27:37 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:43112
I20260928 12:27:37.509111 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9e0d820ac640c689fe138c64cb51c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:27:37 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:46332
I20260928 12:27:37.509266 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9e0d820ac640c689fe138c64cb51c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:27:37 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:46592
I20260928 12:27:37.509840 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 18536
I20260928 12:27:37.511652 18980 tablet_replica.cc:330] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: stopping tablet replica
I20260928 12:27:37.512756 18979 tablet_replica.cc:330] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: stopping tablet replica
I20260928 12:27:37.512938 18980 raft_consensus.cc:2229] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:27:37.513075 18979 raft_consensus.cc:2229] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:27:37.513232 18980 raft_consensus.cc:2258] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:27:37.513401 18979 raft_consensus.cc:2258] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:27:37.514125 18980 ts_tablet_manager.cc:1905] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:27:37.514420 18979 ts_tablet_manager.cc:1905] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:27:37.517619 18980 ts_tablet_manager.cc:1918] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:27:37.520540 18979 ts_tablet_manager.cc:1918] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:27:37.520687 18980 log.cc:1196] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/3f9e0d820ac640c689fe138c64cb51c5
I20260928 12:27:37.520802 18979 log.cc:1196] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/3f9e0d820ac640c689fe138c64cb51c5
I20260928 12:27:37.521142 18979 ts_tablet_manager.cc:1939] T 3f9e0d820ac640c689fe138c64cb51c5 P d7a01ae154b3424184d1299add9e86e5: Deleting consensus metadata
I20260928 12:27:37.521437 18980 ts_tablet_manager.cc:1939] T 3f9e0d820ac640c689fe138c64cb51c5 P 0c78f62d9eef4dc5b9648f8514727436: Deleting consensus metadata
I20260928 12:27:37.521899 18442 catalog_manager.cc:4841] TS 0c78f62d9eef4dc5b9648f8514727436 (127.7.19.66:38781): tablet 3f9e0d820ac640c689fe138c64cb51c5 (table test-table [id=92bfa6554caf462b8a375d202eac75c7]) successfully deleted
I20260928 12:27:37.522403 18439 catalog_manager.cc:4841] TS d7a01ae154b3424184d1299add9e86e5 (127.7.19.67:40581): tablet 3f9e0d820ac640c689fe138c64cb51c5 (table test-table [id=92bfa6554caf462b8a375d202eac75c7]) successfully deleted
W20260928 12:27:37.522666 18442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:34147 (error 108)
I20260928 12:27:37.523330 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 18673
I20260928 12:27:37.530261 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 18810
I20260928 12:27:37.531258 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:27:37.531117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:34147 (local address 127.0.0.1:46598)
0928 12:27:37.531172 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:27:37.531215 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:34147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 12:27:37.531414 18442 catalog_manager.cc:4584] TS b038a4d3496e456089e27feceb544c10 (127.7.19.65:34147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f9e0d820ac640c689fe138c64cb51c5: Network error: Client connection negotiation failed: client connection to 127.7.19.65:34147: connect: Connection refused (error 111)
I20260928 12:27:37.536356 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 18423
I20260928 12:27:37.541756 7245 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:27:37.875972 7245 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:27:37.876070 7245 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:27:38.220197 7245 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:27:38.220 UTC [18009] LOG: received smart shutdown request
2026-09-28 12:27:38.224 UTC [18009] LOG: background worker "logical replication launcher" (PID 18018) exited with exit code 1
2026-09-28 12:27:38.225 UTC [18013] LOG: shutting down
2026-09-28 12:27:38.406 UTC [18009] LOG: database system is shut down
2026-09-28T12:27:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45365 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:27:38 dist-test-slave-cb7x krb5kdc[18991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:27:38 dist-test-slave-cb7x krb5kdc[18991](info): set up 2 sockets
Sep 28 12:27:38 dist-test-slave-cb7x krb5kdc[18991](info): commencing operation
krb5kdc: starting...
W20260928 12:27:39.592829 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:27:39 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:27:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/rangeradmin_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/rangeradmin_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/rangerlookup_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/rangerlookup_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
I20260928 12:27:39.762899 7245 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMIVMOs/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/postgres -l logfile start
2026-09-28 12:27:42.289 UTC [19021] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 12:27:42.289 UTC [19021] LOG: listening on IPv4 address "127.7.19.84", port 55855
2026-09-28 12:27:42.292 UTC [19021] LOG: listening on Unix socket "/tmp/.s.PGSQL.55855"
2026-09-28 12:27:42.313 UTC [19024] LOG: database system was shut down at 2026-09-28 12:27:40 UTC
2026-09-28 12:27:42.317 UTC [19021] LOG: database system is ready to accept connections
I20260928 12:27:43.272534 7245 mini_postgres.cc:96] Postgres bound to 55855
2026-09-28 12:27:43.278 UTC [19032] FATAL: database "slave" does not exist
127.7.19.84:55855 - accepting connections
I20260928 12:27:43.279089 7245 mini_ranger.cc:161] Starting Ranger...
I20260928 12:27:43.293841 7245 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 12:27:43.800383 7245 mini_ranger.cc:87] Created ranger Postgres database
I20260928 12:27:43.800475 7245 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-admin
2026-09-28 12:27:44,081 [I] DB FLAVOR :POSTGRES
2026-09-28 12:27:44,081 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:27:44,082 [I] Checking connection..
2026-09-28 12:27:44,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 12:27:44,312 [I] Checking connection passed.
2026-09-28 12:27:44,312 [I] --------- Verifying version history table ---------
2026-09-28 12:27:44,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:27:44,572 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:27:44,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:27:44,819 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 12:27:44,820 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 12:27:44,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 12:27:45.049 UTC [19096] WARNING: there is no transaction in progress
2026-09-28 12:27:45,055 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 12:27:45,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 12:27:45,308 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 12:27:45,308 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:27:45,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 12:27:45,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 12:27:45,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cb7x.c.gcp-upstream.internal','N') ;"
2026-09-28 12:27:45,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:27:46,247 [I] Table x_portal_user does not exist in database ranger
2026-09-28 12:27:46,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:27:46,477 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 12:27:46,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:27:46,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:27:46,979 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 12:27:46,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 12:27:47.973 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:47.989 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.008 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.023 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.039 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.056 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.112 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.119 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.135 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.145 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.155 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.169 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.179 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.185 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.193 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48.200 UTC [19204] WARNING: there is no transaction in progress
2026-09-28 12:27:48,868 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 12:27:48,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 12:27:49,098 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 12:27:49,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 12:27:49,339 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 12:27:49,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 12:27:49,596 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 12:27:49,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 12:27:49,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 12:27:50,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cb7x.c.gcp-upstream.internal';"
2026-09-28 12:27:50,326 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 12:27:50,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 12:27:50,577 [I] DB_PATCHES have already been applied
I20260928 12:27:50.583688 7245 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMIVMOs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 12:27:50.583755 7245 mini_ranger.cc:193] Using host: 127.7.19.84
I20260928 12:27:50.586793 7245 mini_ranger.cc:227] Ranger admin URL: http://127.7.19.84:46173
Sep 28, 2026 12:27:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:27:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:27:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 12:27:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 12:27:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 12:27:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 12:27:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 12:27:51.722339 7245 mini_ranger.cc:160] Time spent starting Ranger: real 8.443s user 0.000s sys 0.009s
Sep 28, 2026 12:28:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 12:28:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 12:28:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 12:28:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 12:28:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 12:28:12.535663 7245 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.19.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.19.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/rangerkms_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.19.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/rangerkms_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.19.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.19.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.19.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/HTTP_127.7.19.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 12:28:12 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598492, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 12:28:12.627244 7245 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 12:28:12.637357 7245 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 12:28:13.634523 7245 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 12:28:13.634645 7245 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms
I20260928 12:28:13.634668 7245 mini_ranger_kms.cc:227] Using postgres at 127.7.19.84:55855
2026-09-28 12:28:13,794 [I] DB FLAVOR :POSTGRES
2026-09-28 12:28:13,795 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 12:28:13,795 [I] Checking connection
2026-09-28 12:28:13,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 12:28:14,027 [I] connection success
2026-09-28 12:28:14,028 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 12:28:14,028 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 12:28:14,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 12:28:14,273 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 12:28:14,273 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 12:28:14,273 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 12:28:14,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.19.84:55855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 12:28:14,527 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 12:28:15.004931 7245 mini_ranger_kms.cc:310] Created kms service
I20260928 12:28:15.194025 7245 mini_ranger_kms.cc:326] Created kudu user
I20260928 12:28:15.239396 7245 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 12:28:15 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598492, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:15.598839 7245 mini_ranger_kms.cc:384] Added ranger policy
I20260928 12:28:15.598953 7245 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMIVMOs/build/release/bin/postgresql.jar:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMIVMOs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 12:28:15.598984 7245 mini_ranger_kms.cc:240] Using host: 127.7.19.84
I20260928 12:28:15.602571 7245 mini_ranger_kms.cc:276] Ranger KMS PID: 19384
I20260928 12:28:15.602654 7245 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.19.84:36821
Sep 28, 2026 12:28:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 12:28:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 12:28:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 12:28:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 12:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 12:28:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 12:28:17.861308 7245 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.234s user 0.004s sys 0.139s
I20260928 12:28:17.861418 7245 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 12:28:17.861469 7245 mini_ranger_kms.cc:401] 127.7.19.84:36821/kms/v1/keys
Sep 28 12:28:17 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598497, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.19.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 12:28:18 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598497, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.19.84@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
Sep 28 12:28:18 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598497, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.19.84@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
Sep 28 12:28:21 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:28:21.071565 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38557
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:38557
--ranger_config_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:21.193954 19435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:21.194164 19435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:21.194185 19435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:21.195313 19435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:21.195354 19435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:21.195369 19435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:21.195381 19435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:21.195393 19435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:21.197067 19435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.19.84:36821/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:38557
--ranger_config_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:21.197300 19435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:21.197562 19435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:28:21.203509 19435 server_base.cc:1034] running on GCE node
W20260928 12:28:21.203486 19441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:21.203486 19443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:21.203680 19440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:21.204125 19435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:21.204420 19435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:21.205569 19435 hybrid_clock.cc:648] HybridClock initialized: now 1790598501205547 us; error 40 us; skew 500 ppm
Sep 28 12:28:21 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598501, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:21.208256 19435 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:21.208760 19435 webserver.cc:460] Webserver started at http://127.7.19.126:43529/ using document root <none> and password file <none>
I20260928 12:28:21.209016 19435 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:21.209067 19435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:21.209199 19435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:28:21 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598501, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:21.253630 19435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "c52c469981f2470a9e3d131727dec873"
format_stamp: "Formatted at 2026-09-28 12:28:21 on dist-test-slave-cb7x"
server_key: "d28e70bebadfbf68b4e6efda26a5f6d6"
server_key_iv: "36731d7e6a0c449a2cbc6e36293252fa"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:21.254043 19435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "c52c469981f2470a9e3d131727dec873"
format_stamp: "Formatted at 2026-09-28 12:28:21 on dist-test-slave-cb7x"
server_key: "d28e70bebadfbf68b4e6efda26a5f6d6"
server_key_iv: "36731d7e6a0c449a2cbc6e36293252fa"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:21.255501 19435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:28:21.270390 19450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:21.270599 19435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:21.270692 19435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "c52c469981f2470a9e3d131727dec873"
format_stamp: "Formatted at 2026-09-28 12:28:21 on dist-test-slave-cb7x"
server_key: "d28e70bebadfbf68b4e6efda26a5f6d6"
server_key_iv: "36731d7e6a0c449a2cbc6e36293252fa"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:21.270815 19435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:21.287554 19435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:21.289665 19435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:21.289834 19435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:21.294699 19435 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:38557
I20260928 12:28:21.294725 19502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:38557 every 8 connection(s)
I20260928 12:28:21.295094 19435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:21.295754 19503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:21.298017 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 19435
I20260928 12:28:21.298116 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:28:21.298137 19503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Bootstrap starting.
I20260928 12:28:21.298293 7245 external_mini_cluster.cc:1428] Setting key f8a45a9490f595429eccc5f00c8fdcfc
I20260928 12:28:21.298889 19503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:21.299254 19503 log.cc:824] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:21.300021 19503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: No bootstrap required, opened a new log
I20260928 12:28:21.301862 19503 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:21.302030 19503 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:21.302062 19503 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52c469981f2470a9e3d131727dec873, State: Initialized, Role: FOLLOWER
I20260928 12:28:21.302239 19503 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:21.302310 19503 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:21.302336 19503 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:21.302392 19503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:21.302963 19503 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:21.303085 19503 leader_election.cc:302] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52c469981f2470a9e3d131727dec873; no voters:
I20260928 12:28:21.303256 19503 leader_election.cc:288] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:28:21.303741 19503 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:21.303304 19508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:21.304827 19508 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 LEADER]: Becoming Leader. State: Replica: c52c469981f2470a9e3d131727dec873, State: Running, Role: LEADER
I20260928 12:28:21.304960 19508 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
Sep 28 12:28:21 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:21.307080 19509 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c52c469981f2470a9e3d131727dec873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } } }
I20260928 12:28:21.307163 19509 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:21.307209 19503 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:28:21.307529 19517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52c469981f2470a9e3d131727dec873. Latest consensus state: current_term: 1 leader_uuid: "c52c469981f2470a9e3d131727dec873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } } }
I20260928 12:28:21.307682 19517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:21.307600 19523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:21.311048 19523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:21.312969 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:21.298832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.7.19.126:38557)
0928 12:28:21.299104 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:21.299114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:21.299152 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:28:21.299663 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:21.299670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:21.299756 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:21.299905 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:21.300715 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:21.302493 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:21.303539 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:21.304164 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:21.304283 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:21.307188 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:21.307192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:21.307193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:21.307207 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:21.310947 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:21.311361 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:21.311365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:21.311365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:21.311467 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:21.311653 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:21.311654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:21.311654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:21.311985 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:21.312144 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:21.312565 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 12:28:21.312750 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260928 12:28:21.313129 19523 catalog_manager.cc:1347] Generated new cluster ID: 99fe8c48433a440a9f84e17704074160
I20260928 12:28:21.313171 19523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:28:21.348532 19523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:28:21.349407 19523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:21.353616 19523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Generated new TSK 0
I20260928 12:28:21.353796 19523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:28:22.039983 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:38557
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:22.126570 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:22.126897 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:22.126971 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:22.128322 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:22.128368 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:22.128396 19548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:28:22.130018 19548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.19.84:36821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:22.130251 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:22.130498 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:22.135681 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.135696 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.135835 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:22.136149 19548 server_base.cc:1034] running on GCE node
I20260928 12:28:22.136416 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:22.136729 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:22.137900 19548 hybrid_clock.cc:648] HybridClock initialized: now 1790598502137888 us; error 28 us; skew 500 ppm
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:22.140607 19548 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:28:22.141083 19548 webserver.cc:460] Webserver started at http://127.7.19.65:35999/ using document root <none> and password file <none>
I20260928 12:28:22.141330 19548 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:22.141402 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:22.141549 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:22.153164 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "75674ffc11de687dcbe0e37b573e8454"
server_key_iv: "4d439c1466aa741dd8daef9b28f62453"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.153695 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "75674ffc11de687dcbe0e37b573e8454"
server_key_iv: "4d439c1466aa741dd8daef9b28f62453"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.157068 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 12:28:22.168735 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.168984 19548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:22.169080 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "75674ffc11de687dcbe0e37b573e8454"
server_key_iv: "4d439c1466aa741dd8daef9b28f62453"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.169194 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:22.194263 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:22.196768 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:22.196982 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:22.197242 19548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:22.197629 19548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:22.197697 19548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.197746 19548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:22.197783 19548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.205066 19548 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:42925
I20260928 12:28:22.205616 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:28:22.206480 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 19548
I20260928 12:28:22.206564 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:28:22.206745 7245 external_mini_cluster.cc:1428] Setting key 5f4d65d63bf44257e1cac9517d14ae7e
I20260928 12:28:22.213876 19678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:42925 every 8 connection(s)
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
I20260928 12:28:22.227752 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:22.214556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38557 (local address 127.7.19.65:39785)
0928 12:28:22.215187 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:22.215235 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:22.215855 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:22.217508 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:22.217513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:22.217766 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:22.218075 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:22.218083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.219099 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.219100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:22.220365 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:22.220369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.221020 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.221022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:22.221166 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:22.221800 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:22.221810 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:22.223557 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:22.225932 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:22.225940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:22.225943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:22.226254 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:22.226535 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:22.226536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:22.226537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:22.226694 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:22.227007 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:22.227014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:22.227205 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 12:28:22.227322 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 12:28:22.227502 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":84,"threads_started":1}
I20260928 12:28:22.228229 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:22.214846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:39785 (local address 127.7.19.126:38557)
0928 12:28:22.217221 (+ 2375us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:22.217223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:22.217234 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:28:22.217260 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:22.217260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:22.217274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:22.217348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:22.218191 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.218986 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.220473 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.220849 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.220921 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:22.223670 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:22.223671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:22.223672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:22.223681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:22.225827 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:22.226367 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:22.226368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:22.226369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:22.226443 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:22.226764 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:22.226765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:22.226765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:22.226908 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:22.226971 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:22.228034 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0928 12:28:22.228129 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2343,"thread_start_us":108,"threads_started":1}
I20260928 12:28:22.228621 19679 heartbeater.cc:347] Connected to a master server at 127.7.19.126:38557
I20260928 12:28:22.228760 19679 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:22.229005 19679 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:22.229655 19467 ts_manager.cc:194] Registered new tserver with Master: 6ad1350f4aa14c8f97ed7da8c7d0280f (127.7.19.65:42925)
I20260928 12:28:22.230623 19467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:39785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:28:22.244194 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:38557
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:22.354645 19685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:22.354943 19685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:22.354984 19685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:22.356815 19685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:22.356890 19685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:22.356931 19685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:28:22.359189 19685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.19.84:36821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:22.359567 19685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:22.359830 19685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:22.366041 19693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.366225 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.366061 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:22.367035 19685 server_base.cc:1034] running on GCE node
I20260928 12:28:22.367252 19685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:22.367517 19685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:22.368706 19685 hybrid_clock.cc:648] HybridClock initialized: now 1790598502368684 us; error 37 us; skew 500 ppm
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:22.371932 19685 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:28:22.372546 19685 webserver.cc:460] Webserver started at http://127.7.19.66:39795/ using document root <none> and password file <none>
I20260928 12:28:22.372843 19685 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:22.372906 19685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:22.373085 19685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:22.387027 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "3dc56bf8d121445395d037e68d247de1"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "f63a4784cb50d61db992d29229460edf"
server_key_iv: "eecbef7ae665b546ed0cf76ece4bf722"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.387494 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "3dc56bf8d121445395d037e68d247de1"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "f63a4784cb50d61db992d29229460edf"
server_key_iv: "eecbef7ae665b546ed0cf76ece4bf722"
server_key_version: "kuduclusterkey@0"
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:22.390528 19685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 12:28:22.407714 19702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.407953 19685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 12:28:22.408044 19685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "3dc56bf8d121445395d037e68d247de1"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "f63a4784cb50d61db992d29229460edf"
server_key_iv: "eecbef7ae665b546ed0cf76ece4bf722"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.408141 19685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:22.424288 19685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:22.426805 19685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:22.427009 19685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:22.427249 19685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:22.427640 19685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:22.427722 19685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.427780 19685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:22.427824 19685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.435190 19685 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:37543
I20260928 12:28:22.435235 19815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:37543 every 8 connection(s)
I20260928 12:28:22.435637 19685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:22.444875 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:22.436453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:42209 (local address 127.7.19.126:38557)
0928 12:28:22.436529 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:22.436531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:22.436746 (+ 215us) server_negotiation.cc:407] Connection header received
0928 12:28:22.437281 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:22.437283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:22.437298 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:22.437343 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:22.437926 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.438442 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.439610 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.439837 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.439926 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:22.442008 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:22.442009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:22.442010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:22.442018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:22.443438 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:22.443847 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:22.443848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:22.443849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:22.443910 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:22.444177 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:22.444178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:22.444178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:22.444290 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:22.444365 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:22.444648 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 12:28:22.444777 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:28:22.445035 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:22.436426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38557 (local address 127.7.19.66:42209)
0928 12:28:22.436585 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:22.436631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:22.437165 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:22.437451 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:22.437455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:22.437635 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:22.437826 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:22.437830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.438555 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.438556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:22.439520 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:22.439522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.439950 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.439951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:22.440070 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:22.440603 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:22.440611 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:22.441904 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:22.443539 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:22.443546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:22.443548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:22.443769 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:22.443978 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:22.443979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:22.443979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:22.444091 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:22.444358 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:22.444364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:22.444515 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 12:28:22.444627 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:28:22.444789 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 12:28:22.445585 19816 heartbeater.cc:347] Connected to a master server at 127.7.19.126:38557
I20260928 12:28:22.445647 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 19685
I20260928 12:28:22.445704 19816 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:22.445745 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:28:22.445896 19816 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:22.445909 7245 external_mini_cluster.cc:1428] Setting key dc106daee17afc3793b8f8b8036c24f5
I20260928 12:28:22.446595 19466 ts_manager.cc:194] Registered new tserver with Master: 3dc56bf8d121445395d037e68d247de1 (127.7.19.66:37543)
I20260928 12:28:22.447120 19466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:42209
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:28:22.472743 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:38557
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:22.604938 19822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:22.605351 19822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:22.605455 19822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:22.607441 19822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:22.607563 19822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:22.607658 19822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:28:22.610142 19822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.19.84:36821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:22.610674 19822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:22.611038 19822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:22.618355 19828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.618374 19827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.618785 19830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:22.620067 19822 server_base.cc:1034] running on GCE node
I20260928 12:28:22.620283 19822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:22.620545 19822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:22.621735 19822 hybrid_clock.cc:648] HybridClock initialized: now 1790598502621700 us; error 47 us; skew 500 ppm
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:22.625439 19822 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:28:22.626003 19822 webserver.cc:460] Webserver started at http://127.7.19.67:34745/ using document root <none> and password file <none>
I20260928 12:28:22.626302 19822 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:22.626364 19822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:22.626505 19822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:22.649111 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "38c8caac0a6d40318d9a41aea46d81c8"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "2b3827ba1d09b0bdbb9e9bd890337401"
server_key_iv: "6c6f3813a1954eb71e66bcca8714665f"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.649739 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "38c8caac0a6d40318d9a41aea46d81c8"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "2b3827ba1d09b0bdbb9e9bd890337401"
server_key_iv: "6c6f3813a1954eb71e66bcca8714665f"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.651861 19822 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 12:28:22.669922 19837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.670154 19822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:28:22.670248 19822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "38c8caac0a6d40318d9a41aea46d81c8"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "2b3827ba1d09b0bdbb9e9bd890337401"
server_key_iv: "6c6f3813a1954eb71e66bcca8714665f"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.670344 19822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:22.696849 19822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:22.699517 19822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:22.699735 19822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:22.700017 19822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:22.700479 19822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:22.700577 19822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.700656 19822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:22.700766 19822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.707384 19822 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:37991
I20260928 12:28:22.707656 19950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:37991 every 8 connection(s)
I20260928 12:28:22.707921 19822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:28:22.710315 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 19822
I20260928 12:28:22.710410 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:28:22.710590 7245 external_mini_cluster.cc:1428] Setting key 01120d9037239a9791b4b1f2ba195e2b
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:22.719585 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:22.708701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:58123 (local address 127.7.19.126:38557)
0928 12:28:22.708760 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:22.708763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:22.709022 (+ 259us) server_negotiation.cc:407] Connection header received
0928 12:28:22.709690 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:22.709691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:22.709704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:22.709779 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:22.710497 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.711248 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.712645 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.712991 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.713071 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:22.715699 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:22.715700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:22.715701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:22.715709 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:22.717823 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:22.718383 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:22.718385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:22.718385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:22.718452 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:22.718761 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:22.718762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:22.718762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:22.718918 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:22.718966 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:22.719390 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 12:28:22.719482 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:28:22.720981 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:22.708706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38557 (local address 127.7.19.67:58123)
0928 12:28:22.708895 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:22.708938 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:22.709566 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:22.709850 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:22.709854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:22.710093 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:22.710392 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:22.710398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.711362 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.711363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:22.712554 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:22.712557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:22.713091 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:22.713093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:22.713236 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:22.713932 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:22.713942 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:22.715582 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:22.717949 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:22.717956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:22.717959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:22.718279 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:22.718532 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:22.718532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:22.718533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:22.718678 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:22.719021 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:22.719028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:22.719219 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 12:28:22.719736 (+ 517us) client_negotiation.cc:240] Negotiation successful
0928 12:28:22.720712 (+ 976us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260928 12:28:22.722404 19951 heartbeater.cc:347] Connected to a master server at 127.7.19.126:38557
I20260928 12:28:22.722573 19951 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:22.723037 19951 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:22.723664 19466 ts_manager.cc:194] Registered new tserver with Master: 38c8caac0a6d40318d9a41aea46d81c8 (127.7.19.67:37991)
I20260928 12:28:22.724160 19466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:58123
I20260928 12:28:22.733285 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:28:22.733999 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 19548
I20260928 12:28:22.749405 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 19685
I20260928 12:28:22.757236 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 19822
I20260928 12:28:22.767971 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 19435
I20260928 12:28:22.774878 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38557
--webserver_interface=127.7.19.126
--webserver_port=43529
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:38557
--ranger_config_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:22.877730 19954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:22.877942 19954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:22.877964 19954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:22.879180 19954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:22.879227 19954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:22.879241 19954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:22.879254 19954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:22.879266 19954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:22.880960 19954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:38557
--ranger_config_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:22.881227 19954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:22.881456 19954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:22.887238 19960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:22.887306 19954 server_base.cc:1034] running on GCE node
W20260928 12:28:22.887382 19962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:22.887512 19959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:22.887758 19954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:22.888092 19954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:22.889261 19954 hybrid_clock.cc:648] HybridClock initialized: now 1790598502889250 us; error 31 us; skew 500 ppm
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:22.900192 19954 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:22.900799 19954 webserver.cc:460] Webserver started at http://127.7.19.126:43529/ using document root <none> and password file <none>
I20260928 12:28:22.901059 19954 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:22.901142 19954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:28:22 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598502, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:22.947511 19954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:22.948276 19970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:22.948535 19954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:22.948655 19954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "c52c469981f2470a9e3d131727dec873"
format_stamp: "Formatted at 2026-09-28 12:28:21 on dist-test-slave-cb7x"
server_key: "d28e70bebadfbf68b4e6efda26a5f6d6"
server_key_iv: "36731d7e6a0c449a2cbc6e36293252fa"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:22.949198 19954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:22.963366 19954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:22.965090 19954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:22.965209 19954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:22.969604 19954 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:38557
I20260928 12:28:22.969645 20022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:38557 every 8 connection(s)
I20260928 12:28:22.970036 19954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:22.971179 20023 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:28:22.971630 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 19954
I20260928 12:28:22.971899 20023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Bootstrap starting.
I20260928 12:28:22.971909 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:42925
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:38557
--webserver_port=35999
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:22.979872 20023 log.cc:824] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:22.981619 20023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:28:22.981899 20023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Bootstrap complete.
I20260928 12:28:22.984180 20023 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:22.984431 20023 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52c469981f2470a9e3d131727dec873, State: Initialized, Role: FOLLOWER
I20260928 12:28:22.984642 20023 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:22.984748 20023 raft_consensus.cc:388] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:22.984800 20023 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:22.984848 20023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:28:22.985930 20023 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:22.986048 20023 leader_election.cc:302] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c52c469981f2470a9e3d131727dec873; no voters:
I20260928 12:28:22.986256 20023 leader_election.cc:288] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:28:22.986289 20027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:28:22.986387 20027 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [term 2 LEADER]: Becoming Leader. State: Replica: c52c469981f2470a9e3d131727dec873, State: Running, Role: LEADER
I20260928 12:28:22.986507 20023 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:22.986490 20027 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } }
I20260928 12:28:22.986840 20028 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c52c469981f2470a9e3d131727dec873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } } }
I20260928 12:28:22.986887 20029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52c469981f2470a9e3d131727dec873. Latest consensus state: current_term: 2 leader_uuid: "c52c469981f2470a9e3d131727dec873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c52c469981f2470a9e3d131727dec873" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38557 } } }
I20260928 12:28:22.987020 20029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:22.987222 20028 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:22.987376 20035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:22.987979 20035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:22.988174 20035 catalog_manager.cc:1259] Loaded cluster ID: 99fe8c48433a440a9f84e17704074160
I20260928 12:28:22.988250 20035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:28:22.988442 20023 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 12:28:22.989750 20035 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:22.989979 20035 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Loaded TSK: 0
I20260928 12:28:22.990242 20035 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:28:23.089156 20025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:23.089411 20025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:23.089454 20025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:23.091235 20025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:23.091292 20025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:23.091332 20025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:28:23.093560 20025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:42925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=35999
--tserver_master_addrs=127.7.19.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:23.093784 20025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:23.094027 20025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:23.100103 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:23.100132 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:23.100452 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:23.101198 20025 server_base.cc:1034] running on GCE node
I20260928 12:28:23.101522 20025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:23.101778 20025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:23.109733 20025 hybrid_clock.cc:648] HybridClock initialized: now 1790598503109706 us; error 38 us; skew 500 ppm
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:23.113131 20025 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:28:23.113615 20025 webserver.cc:460] Webserver started at http://127.7.19.65:35999/ using document root <none> and password file <none>
I20260928 12:28:23.113873 20025 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:23.113940 20025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:23.139989 20025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:23.141433 20071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.141606 20025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:23.141695 20025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "75674ffc11de687dcbe0e37b573e8454"
server_key_iv: "4d439c1466aa741dd8daef9b28f62453"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:23.142177 20025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:23.159837 20025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:23.162024 20025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:23.162165 20025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:23.162413 20025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:23.162776 20025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:23.162811 20025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.162835 20025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:23.162847 20025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.170011 20025 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:42925
I20260928 12:28:23.170248 20184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:42925 every 8 connection(s)
I20260928 12:28:23.170449 20025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:28:23.172577 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20025
I20260928 12:28:23.172906 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:37543
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:38557
--webserver_port=39795
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:23.186882 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.171280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:37169 (local address 127.7.19.126:38557)
0928 12:28:23.171539 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:23.171547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:23.171585 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:28:23.172142 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:23.172147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:23.172190 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:23.172328 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:23.173137 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.174033 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.175291 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.175710 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.175825 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.179436 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:23.179440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:23.179441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:23.179452 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:23.184536 (+ 5084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.185044 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.185047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.185048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.185159 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.185462 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.185463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.185463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.185755 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:23.185833 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:23.186556 (+ 723us) server_negotiation.cc:299] Negotiation successful
0928 12:28:23.186678 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260928 12:28:23.187278 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.171168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38557 (local address 127.7.19.65:37169)
0928 12:28:23.171344 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:23.171386 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:23.171914 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:23.172364 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:23.172369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:23.172567 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:23.172856 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:23.172862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.174156 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.174158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:23.175202 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:23.175205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.175824 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.175826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:23.175961 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.176505 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:23.176515 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:23.179301 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:23.184647 (+ 5346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:23.184653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:23.184655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:23.184948 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:23.185239 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:23.185240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:23.185241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:23.185370 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:23.185861 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:23.185867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:23.186013 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 12:28:23.186109 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 12:28:23.186256 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260928 12:28:23.188452 20185 heartbeater.cc:347] Connected to a master server at 127.7.19.126:38557
I20260928 12:28:23.188544 20185 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.188746 20185 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
W20260928 12:28:23.189638 20185 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:38557 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:28:23.189704 20185 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.189760 20185 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:23.190148 20185 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.190228 20185 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
W20260928 12:28:23.190490 20185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:28:23.294065 20189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:23.294360 20189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:23.294401 20189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:23.296205 20189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:23.296272 20189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:23.296319 20189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:28:23.298718 20189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:37543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=39795
--tserver_master_addrs=127.7.19.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:23.298974 20189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:23.299233 20189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:23.304926 20195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:23.305749 20194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:23.306128 20197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:23.306149 20189 server_base.cc:1034] running on GCE node
I20260928 12:28:23.306450 20189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:23.306721 20189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:23.307898 20189 hybrid_clock.cc:648] HybridClock initialized: now 1790598503307884 us; error 27 us; skew 500 ppm
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:23.311178 20189 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:28:23.311677 20189 webserver.cc:460] Webserver started at http://127.7.19.66:39795/ using document root <none> and password file <none>
I20260928 12:28:23.311959 20189 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:23.312048 20189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:23.331156 20189 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:23.331903 20205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.332083 20189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:23.332171 20189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "3dc56bf8d121445395d037e68d247de1"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "f63a4784cb50d61db992d29229460edf"
server_key_iv: "eecbef7ae665b546ed0cf76ece4bf722"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:23.333588 20189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:23.350323 20189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:23.352530 20189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:23.352753 20189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:23.353032 20189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:23.353428 20189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:23.353505 20189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.353549 20189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:23.353582 20189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.361415 20189 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:37543
I20260928 12:28:23.361833 20189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:28:23.362517 20318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:37543 every 8 connection(s)
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:23.369827 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20189
I20260928 12:28:23.370117 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.19.84:36821/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:37991
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:38557
--webserver_port=34745
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:23.373226 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.363039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:46885 (local address 127.7.19.126:38557)
0928 12:28:23.363093 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:23.363096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:23.363108 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:28:23.363672 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:23.363674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:23.363690 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:23.363757 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:23.364542 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.365345 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.366816 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.367132 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.367214 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.369624 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:23.369626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:23.369626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:23.369634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:23.371294 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.371892 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.371893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.371894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.371961 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.372271 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.372272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.372272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.372428 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:23.372483 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:23.372956 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 12:28:23.373067 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:28:23.373850 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.362715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38557 (local address 127.7.19.66:46885)
0928 12:28:23.362895 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:23.362936 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:23.363552 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:23.363829 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:23.363833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:23.364107 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:23.364425 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:23.364432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.365481 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.365482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:23.366714 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:23.366717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.367265 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.367267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:23.367402 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.367970 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:23.367979 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:23.369513 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:23.371428 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:23.371435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:23.371438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:23.371780 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:23.372048 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:23.372049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:23.372049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:23.372191 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:23.372541 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:23.372547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:23.372769 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 12:28:23.373379 (+ 610us) client_negotiation.cc:240] Negotiation successful
0928 12:28:23.373553 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260928 12:28:23.374421 20319 heartbeater.cc:347] Connected to a master server at 127.7.19.126:38557
I20260928 12:28:23.374593 20319 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.374796 20319 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
W20260928 12:28:23.375582 20319 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:38557 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:28:23.375658 20319 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.375717 20319 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:23.376111 20319 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.376209 20319 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
W20260928 12:28:23.376611 20319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:28:23.490875 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:23.491187 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:23.491236 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:23.493217 20322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:23.493280 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:23.493324 20322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:28:23.495810 20322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.19.84:36821/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=34745
--tserver_master_addrs=127.7.19.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:23.496104 20322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:23.496381 20322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:23.502718 20331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:23.502848 20328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:23.503051 20329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:23.503186 20322 server_base.cc:1034] running on GCE node
I20260928 12:28:23.503413 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:23.503703 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:23.504881 20322 hybrid_clock.cc:648] HybridClock initialized: now 1790598503504851 us; error 44 us; skew 500 ppm
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:23.509393 20322 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:28:23.509907 20322 webserver.cc:460] Webserver started at http://127.7.19.67:34745/ using document root <none> and password file <none>
I20260928 12:28:23.510190 20322 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:23.510272 20322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:23.529057 20322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 12:28:23.529979 20339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.530179 20322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:23.530272 20322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "38c8caac0a6d40318d9a41aea46d81c8"
format_stamp: "Formatted at 2026-09-28 12:28:22 on dist-test-slave-cb7x"
server_key: "2b3827ba1d09b0bdbb9e9bd890337401"
server_key_iv: "6c6f3813a1954eb71e66bcca8714665f"
server_key_version: "kuduclusterkey@0"
I20260928 12:28:23.530654 20322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:23.579427 20322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:23.581692 20322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:23.581840 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:23.582104 20322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:23.582459 20322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:23.582522 20322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.582566 20322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:23.582592 20322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:23.591010 20322 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:37991
I20260928 12:28:23.591080 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:37991 every 8 connection(s)
I20260928 12:28:23.591414 20322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:28:23.598939 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20322
Sep 28 12:28:23 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598503, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:23.602856 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.592470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:58749 (local address 127.7.19.126:38557)
0928 12:28:23.592550 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:23.592552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:23.592725 (+ 173us) server_negotiation.cc:407] Connection header received
0928 12:28:23.593400 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:23.593402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:23.593414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:23.593505 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:23.594193 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.594946 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.596429 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.596746 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.596820 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.599604 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:23.599606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:23.599606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:23.599614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:23.601280 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.601780 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.601782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.601782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.601847 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.602138 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.602139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.602139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.602264 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:23.602335 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:23.602685 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 12:28:23.602768 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:28:23.603031 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.592204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:38557 (local address 127.7.19.67:58749)
0928 12:28:23.592573 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:23.592620 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:23.593272 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:23.593583 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:23.593588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:23.593809 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:23.594084 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:23.594090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.595059 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.595060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:23.596330 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:23.596332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.596919 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.596920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:23.597066 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.597686 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:23.597696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:23.599401 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:23.601371 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:23.601377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:23.601380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:23.601700 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:23.601923 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:23.601924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:23.601924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:23.602064 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:23.602344 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:23.602350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:23.602523 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 12:28:23.602651 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 12:28:23.602804 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":84,"threads_started":1}
I20260928 12:28:23.603487 20453 heartbeater.cc:347] Connected to a master server at 127.7.19.126:38557
I20260928 12:28:23.603603 20453 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.603809 20453 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
W20260928 12:28:23.604566 20453 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:38557 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 12:28:23.604638 20453 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.604744 20453 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:23.605090 20453 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:23.605170 20453 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
W20260928 12:28:23.605476 20453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:28:23.607604 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:23.599327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.7.19.126:38557)
0928 12:28:23.599667 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:23.599669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:23.599676 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:28:23.599693 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:23.599694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:23.599704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:23.599777 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:23.600638 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.601456 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.602420 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:23.602666 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:23.602722 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:23.604532 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:23.604533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:23.604534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:23.604541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:23.606486 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.606818 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.606819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.606820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.606881 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:23.607068 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:23.607069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:23.607070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:23.607211 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:23.607378 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:23.607417 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:28:23.607506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":67,"threads_started":1}
Sep 28 12:28:24 dist-test-slave-cb7x krb5kdc[18991](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790598504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:24.190685 20185 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:24.190848 20185 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:24.191241 19987 ts_manager.cc:194] Registered new tserver with Master: 6ad1350f4aa14c8f97ed7da8c7d0280f (127.7.19.65:42925)
I20260928 12:28:24.192276 19987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:37169
I20260928 12:28:24.376878 20319 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:24.377044 20319 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:24.377427 19987 ts_manager.cc:194] Registered new tserver with Master: 3dc56bf8d121445395d037e68d247de1 (127.7.19.66:37543)
I20260928 12:28:24.378027 19987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:46885
Sep 28 12:28:24 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:24.605656 20453 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:24.605809 20453 heartbeater.cc:511] Master 127.7.19.126:38557 requested a full tablet report, sending...
I20260928 12:28:24.606180 19987 ts_manager.cc:194] Registered new tserver with Master: 38c8caac0a6d40318d9a41aea46d81c8 (127.7.19.67:37991)
I20260928 12:28:24.606690 19987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:58749
I20260928 12:28:24.608021 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:28:24.616869 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.608960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.7.19.126:38557)
0928 12:28:24.609280 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.609282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.609289 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:28:24.609309 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.609309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.609323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.609380 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:24.610055 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.610810 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.611831 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.612146 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.612202 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.613736 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:24.613737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:24.613737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:24.613745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:24.615417 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:24.615788 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:24.615789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:24.615790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:24.615845 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:24.616038 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:24.616038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:24.616039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:24.616157 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:24.616245 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.616660 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.616767 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":63,"threads_started":1}
I20260928 12:28:24.619288 19987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:28:24.621497 19987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:28:24.637470 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.631162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:42925 (local address 127.0.0.1:43550)
0928 12:28:24.632366 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.632374 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:24.632406 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:24.632788 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:24.632791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:24.632793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:24.632861 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.632862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.634477 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.634478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.635755 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.635758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.637161 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.637162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.637233 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.637246 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:28:24.637312 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 12:28:24.637359 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":53,"threads_started":1}
I20260928 12:28:24.637789 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.631332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.7.19.65:42925)
0928 12:28:24.632513 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.632517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.632528 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:28:24.632563 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.632566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.632582 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.632654 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:24.632950 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.634348 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.635847 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.636983 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.637064 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.637128 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.637669 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.637708 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":38,"threads_started":1}
I20260928 12:28:24.638446 20119 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1f4831bb245aa86aa96a15b446f3e (DEFAULT_TABLE table=test-table [id=c27d00c1e4494b5db5b3e016e509c497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:24.638820 20119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f1f4831bb245aa86aa96a15b446f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:24.641939 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.631466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37543 (local address 127.0.0.1:37760)
0928 12:28:24.634864 (+ 3398us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.634874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:24.634932 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:24.636969 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:24.636970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:24.636973 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:24.637015 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.637016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.638258 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.638259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.639284 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.639287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.641706 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.641707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.641770 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.641775 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:28:24.641832 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:28:24.641868 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3382,"thread_start_us":94,"threads_started":1}
I20260928 12:28:24.642215 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.631687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.7.19.66:37543)
0928 12:28:24.632104 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.632109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.635004 (+ 2895us) server_negotiation.cc:407] Connection header received
0928 12:28:24.635392 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.635395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.635411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.635477 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:24.637094 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.638171 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.639379 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.640083 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.640152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.640207 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.642093 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.642141 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":53,"threads_started":1}
I20260928 12:28:24.642916 20486 tablet_bootstrap.cc:492] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: Bootstrap starting.
I20260928 12:28:24.642818 20253 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1f4831bb245aa86aa96a15b446f3e (DEFAULT_TABLE table=test-table [id=c27d00c1e4494b5db5b3e016e509c497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:24.643324 20253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f1f4831bb245aa86aa96a15b446f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:24.643549 20486 tablet_bootstrap.cc:654] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:24.643852 20486 log.cc:824] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:24.644904 20486 tablet_bootstrap.cc:492] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: No bootstrap required, opened a new log
I20260928 12:28:24.644979 20486 ts_tablet_manager.cc:1397] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:24.647526 20486 raft_consensus.cc:348] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.647697 20486 raft_consensus.cc:374] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:24.647740 20486 raft_consensus.cc:729] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad1350f4aa14c8f97ed7da8c7d0280f, State: Initialized, Role: FOLLOWER
I20260928 12:28:24.647843 20486 consensus_queue.cc:260] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.648095 20486 ts_tablet_manager.cc:1428] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:28:24.648571 20185 heartbeater.cc:503] Master 127.7.19.126:38557 was elected leader, sending a full tablet report...
I20260928 12:28:24.649466 20488 tablet_bootstrap.cc:492] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Bootstrap starting.
I20260928 12:28:24.650063 20488 tablet_bootstrap.cc:654] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:24.650367 20488 log.cc:824] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:24.651535 20488 tablet_bootstrap.cc:492] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: No bootstrap required, opened a new log
I20260928 12:28:24.651604 20488 ts_tablet_manager.cc:1397] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:24.654325 20488 raft_consensus.cc:348] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.654496 20488 raft_consensus.cc:374] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:24.654541 20488 raft_consensus.cc:729] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc56bf8d121445395d037e68d247de1, State: Initialized, Role: FOLLOWER
I20260928 12:28:24.654704 20488 consensus_queue.cc:260] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.655004 20488 ts_tablet_manager.cc:1428] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:28:24.655139 20319 heartbeater.cc:503] Master 127.7.19.126:38557 was elected leader, sending a full tablet report...
I20260928 12:28:24.659634 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.630992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.7.19.67:37991)
0928 12:28:24.640283 (+ 9291us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.640288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.640299 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:28:24.640334 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.640336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.640354 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.640598 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:24.640696 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.656884 (+ 16188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.658306 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.659142 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.659193 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.659252 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.659502 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.659547 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9255,"thread_start_us":46,"threads_started":1}
I20260928 12:28:24.660197 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.630917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:37991 (local address 127.0.0.1:47494)
0928 12:28:24.632167 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.632207 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:24.632269 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:24.640453 (+ 8184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:24.640454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:24.640456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:24.640512 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.640514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.656961 (+ 16447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.656963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.658184 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.658188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.659330 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.659331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.659405 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.659409 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:28:24.660066 (+ 657us) client_negotiation.cc:240] Negotiation successful
0928 12:28:24.660116 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"thread_start_us":73,"threads_started":1}
I20260928 12:28:24.660979 20373 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1f4831bb245aa86aa96a15b446f3e (DEFAULT_TABLE table=test-table [id=c27d00c1e4494b5db5b3e016e509c497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:24.661365 20373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f1f4831bb245aa86aa96a15b446f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:24.665325 20492 tablet_bootstrap.cc:492] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Bootstrap starting.
I20260928 12:28:24.665975 20492 tablet_bootstrap.cc:654] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:24.666304 20492 log.cc:824] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:24.666996 20492 tablet_bootstrap.cc:492] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: No bootstrap required, opened a new log
I20260928 12:28:24.667074 20492 ts_tablet_manager.cc:1397] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:24.669996 20492 raft_consensus.cc:348] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.670188 20492 raft_consensus.cc:374] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:24.670228 20492 raft_consensus.cc:729] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c8caac0a6d40318d9a41aea46d81c8, State: Initialized, Role: FOLLOWER
I20260928 12:28:24.670372 20492 consensus_queue.cc:260] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.670626 20492 ts_tablet_manager.cc:1428] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 12:28:24.671140 20453 heartbeater.cc:503] Master 127.7.19.126:38557 was elected leader, sending a full tablet report...
W20260928 12:28:24.671811 20186 tablet.cc:2369] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 12:28:24 dist-test-slave-cb7x krb5kdc[18991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598504, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for HTTP/127.7.19.84@KRBTEST.COM
I20260928 12:28:24.774068 20489 raft_consensus.cc:482] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:24.774225 20489 raft_consensus.cc:504] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.774663 20489 leader_election.cc:288] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dc56bf8d121445395d037e68d247de1 (127.7.19.66:37543), 38c8caac0a6d40318d9a41aea46d81c8 (127.7.19.67:37991)
I20260928 12:28:24.783218 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.775063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:58993 (local address 127.7.19.66:37543)
0928 12:28:24.775122 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.775125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.775189 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:28:24.775288 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.775289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.775301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.775329 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:24.775899 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.776699 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.782068 (+ 5369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.782674 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.782745 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.782757 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.783096 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.783144 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 12:28:24.783887 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.774849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37543 (local address 127.7.19.65:58993)
0928 12:28:24.775129 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.775140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:24.775216 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:24.775734 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:24.775735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:24.775737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:24.775820 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.775822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.776804 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.776805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.781986 (+ 5181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.781991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.782932 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.782933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.783005 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.783009 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:28:24.783738 (+ 729us) client_negotiation.cc:240] Negotiation successful
0928 12:28:24.783787 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260928 12:28:24.784240 20273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1f4831bb245aa86aa96a15b446f3e" candidate_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc56bf8d121445395d037e68d247de1" is_pre_election: true
I20260928 12:28:24.784423 20273 raft_consensus.cc:2454] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad1350f4aa14c8f97ed7da8c7d0280f in term 0.
I20260928 12:28:24.784731 20072 leader_election.cc:302] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc56bf8d121445395d037e68d247de1, 6ad1350f4aa14c8f97ed7da8c7d0280f; no voters:
I20260928 12:28:24.784874 20489 raft_consensus.cc:2781] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:28:24.784929 20489 raft_consensus.cc:482] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:28:24.784958 20489 raft_consensus.cc:3037] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:24.785712 20489 raft_consensus.cc:504] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.785909 20489 leader_election.cc:288] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [CANDIDATE]: Term 1 election: Requested vote from peers 3dc56bf8d121445395d037e68d247de1 (127.7.19.66:37543), 38c8caac0a6d40318d9a41aea46d81c8 (127.7.19.67:37991)
I20260928 12:28:24.786154 20273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1f4831bb245aa86aa96a15b446f3e" candidate_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc56bf8d121445395d037e68d247de1"
I20260928 12:28:24.786240 20273 raft_consensus.cc:3037] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:24.787055 20273 raft_consensus.cc:2454] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad1350f4aa14c8f97ed7da8c7d0280f in term 1.
I20260928 12:28:24.787241 20072 leader_election.cc:302] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc56bf8d121445395d037e68d247de1, 6ad1350f4aa14c8f97ed7da8c7d0280f; no voters:
I20260928 12:28:24.787328 20489 raft_consensus.cc:2781] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:24.787478 20489 raft_consensus.cc:686] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [term 1 LEADER]: Becoming Leader. State: Replica: 6ad1350f4aa14c8f97ed7da8c7d0280f, State: Running, Role: LEADER
I20260928 12:28:24.787582 20489 consensus_queue.cc:237] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } }
I20260928 12:28:24.788393 19987 catalog_manager.cc:5495] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad1350f4aa14c8f97ed7da8c7d0280f (127.7.19.65). New cstate: current_term: 1 leader_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42925 } health_report { overall_health: HEALTHY } } }
I20260928 12:28:24.789520 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.775066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:38217 (local address 127.7.19.67:37991)
0928 12:28:24.775154 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.775156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.783461 (+ 8305us) server_negotiation.cc:407] Connection header received
0928 12:28:24.783577 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.783578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.783593 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.783622 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:24.784540 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.785304 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.787856 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.788400 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.788462 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.788472 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.789393 (+ 921us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.789454 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 12:28:24.789829 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.774860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:37991 (local address 127.7.19.65:38217)
0928 12:28:24.783398 (+ 8538us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.783410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:24.783490 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:24.783685 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:24.783686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:24.783687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:24.783933 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.783936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.785400 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.785400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.787075 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:24.787078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.789239 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.789239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:24.789311 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.789315 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:28:24.789720 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 12:28:24.789759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8516,"thread_start_us":107,"threads_started":1}
I20260928 12:28:24.790583 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1f4831bb245aa86aa96a15b446f3e" candidate_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c8caac0a6d40318d9a41aea46d81c8" is_pre_election: true
I20260928 12:28:24.790587 20407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f1f4831bb245aa86aa96a15b446f3e" candidate_uuid: "6ad1350f4aa14c8f97ed7da8c7d0280f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c8caac0a6d40318d9a41aea46d81c8"
I20260928 12:28:24.790740 20392 raft_consensus.cc:2454] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad1350f4aa14c8f97ed7da8c7d0280f in term 0.
I20260928 12:28:24.825670 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.822285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.7.19.65:42925)
0928 12:28:24.822331 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:24.822333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:24.822341 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:28:24.822383 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:24.822384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:24.822407 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:24.822486 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:28:24.822706 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.823443 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.824564 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:24.824884 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:24.824942 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:24.825137 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:28:24.825272 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:28:24.825344 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:24.825569 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 12:28:24.825590 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:28:24.827680 20407 raft_consensus.cc:3037] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:24.828622 20407 raft_consensus.cc:1261] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 1 FOLLOWER]: Refusing update from remote peer 6ad1350f4aa14c8f97ed7da8c7d0280f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:24.828955 20273 raft_consensus.cc:1261] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 1 FOLLOWER]: Refusing update from remote peer 6ad1350f4aa14c8f97ed7da8c7d0280f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:24.829319 20489 consensus_queue.cc:1035] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c8caac0a6d40318d9a41aea46d81c8" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:24.829478 20489 consensus_queue.cc:1035] T 11f1f4831bb245aa86aa96a15b446f3e P 6ad1350f4aa14c8f97ed7da8c7d0280f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc56bf8d121445395d037e68d247de1" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:24.835291 20502 mvcc.cc:205] Tried to move back new op lower bound from 7334291475770982400 to 7334291475610529792. Current Snapshot: MvccSnapshot[applied={T|T < 7334291475770982400}]
I20260928 12:28:24.835976 20505 mvcc.cc:205] Tried to move back new op lower bound from 7334291475770982400 to 7334291475610529792. Current Snapshot: MvccSnapshot[applied={T|T < 7334291475770982400}]
I20260928 12:28:24.838418 20503 mvcc.cc:205] Tried to move back new op lower bound from 7334291475770982400 to 7334291475610529792. Current Snapshot: MvccSnapshot[applied={T|T < 7334291475770982400}]
I20260928 12:28:24.841246 19986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:24.841342 19986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:24.841986 19986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c52c469981f2470a9e3d131727dec873: Sending DeleteTablet for 3 replicas of tablet 11f1f4831bb245aa86aa96a15b446f3e
I20260928 12:28:24.842478 20373 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f1f4831bb245aa86aa96a15b446f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:24 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:47494
I20260928 12:28:24.842681 20119 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f1f4831bb245aa86aa96a15b446f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:24 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:43550
I20260928 12:28:24.843008 20253 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f1f4831bb245aa86aa96a15b446f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:24 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:37760
I20260928 12:28:24.843705 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20025
I20260928 12:28:24.845314 20514 tablet_replica.cc:330] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: stopping tablet replica
I20260928 12:28:24.845412 20514 raft_consensus.cc:2229] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:24.845511 20514 raft_consensus.cc:2258] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:24.845925 20514 ts_tablet_manager.cc:1905] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:24.847383 20512 tablet_replica.cc:330] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: stopping tablet replica
I20260928 12:28:24.847476 20512 raft_consensus.cc:2229] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:24.847580 20512 raft_consensus.cc:2258] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:24.847956 20512 ts_tablet_manager.cc:1905] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:24.849560 20512 ts_tablet_manager.cc:1918] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:24.849633 20512 log.cc:1196] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/11f1f4831bb245aa86aa96a15b446f3e
I20260928 12:28:24.849890 20512 ts_tablet_manager.cc:1939] T 11f1f4831bb245aa86aa96a15b446f3e P 38c8caac0a6d40318d9a41aea46d81c8: Deleting consensus metadata
I20260928 12:28:24.850262 19973 catalog_manager.cc:4841] TS 38c8caac0a6d40318d9a41aea46d81c8 (127.7.19.67:37991): tablet 11f1f4831bb245aa86aa96a15b446f3e (table test-table [id=c27d00c1e4494b5db5b3e016e509c497]) successfully deleted
I20260928 12:28:24.853572 20514 ts_tablet_manager.cc:1918] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:24.853699 20514 log.cc:1196] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/11f1f4831bb245aa86aa96a15b446f3e
I20260928 12:28:24.854009 20514 ts_tablet_manager.cc:1939] T 11f1f4831bb245aa86aa96a15b446f3e P 3dc56bf8d121445395d037e68d247de1: Deleting consensus metadata
W20260928 12:28:24.854483 19972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:42925 (error 108)
I20260928 12:28:24.855352 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.855232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:42925 (local address 127.0.0.1:43556)
0928 12:28:24.855277 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.855316 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:42925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 12:28:24.855363 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20189
W20260928 12:28:24.855479 19972 catalog_manager.cc:4584] TS 6ad1350f4aa14c8f97ed7da8c7d0280f (127.7.19.65:42925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11f1f4831bb245aa86aa96a15b446f3e: Network error: Client connection negotiation failed: client connection to 127.7.19.65:42925: connect: Connection refused (error 111)
I20260928 12:28:24.864310 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:24.864196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37543 (local address 127.0.0.1:37764)
0928 12:28:24.864246 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:24.864269 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.66:37543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 12:28:24.864905 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20322
I20260928 12:28:24.871737 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 19954
I20260928 12:28:24.879056 7245 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 12:28:25.223492 7245 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 12:28:25.223588 7245 mini_ranger.cc:66] Stopping Ranger...
I20260928 12:28:25.567879 7245 mini_ranger.cc:68] Stopped Ranger
2026-09-28 12:28:25.568 UTC [19021] LOG: received smart shutdown request
2026-09-28 12:28:25.572 UTC [19021] LOG: background worker "logical replication launcher" (PID 19030) exited with exit code 1
2026-09-28 12:28:25.572 UTC [19025] LOG: shutting down
2026-09-28 12:28:25.758 UTC [19021] LOG: database system is shut down
2026-09-28T12:28:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47333 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:28:25 dist-test-slave-cb7x krb5kdc[20524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:28:25 dist-test-slave-cb7x krb5kdc[20524](info): set up 2 sockets
Sep 28 12:28:25 dist-test-slave-cb7x krb5kdc[20524](info): commencing operation
krb5kdc: starting...
W20260928 12:28:26.956552 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:28:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:28:27.075905 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41205
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:41205,127.7.19.125:34363,127.7.19.124:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:27.156451 20537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:27.156651 20537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:27.156688 20537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:27.157863 20537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:27.157907 20537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:27.157923 20537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:27.157936 20537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:27.157948 20537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:27.159329 20537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:41205,127.7.19.125:34363,127.7.19.124:40917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:27.159552 20537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:27.159721 20537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:28:27.164530 20537 server_base.cc:1034] running on GCE node
W20260928 12:28:27.164476 20543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:27.164471 20542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:27.164471 20545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:27.165035 20537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:27.165365 20537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:27.166553 20537 hybrid_clock.cc:648] HybridClock initialized: now 1790598507166535 us; error 42 us; skew 500 ppm
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:27.169071 20537 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:27.169516 20537 webserver.cc:460] Webserver started at http://127.7.19.126:35173/ using document root <none> and password file <none>
I20260928 12:28:27.169744 20537 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:27.169811 20537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:27.169975 20537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:27.171043 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "3a512c713b735ab835f26d779410e4c5"
server_key_iv: "3df618f821a3024c7910b6f43d59a83a"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.171413 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "3a512c713b735ab835f26d779410e4c5"
server_key_iv: "3df618f821a3024c7910b6f43d59a83a"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.172765 20537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:28:27.173545 20552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:27.173769 20537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:27.173875 20537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "3a512c713b735ab835f26d779410e4c5"
server_key_iv: "3df618f821a3024c7910b6f43d59a83a"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.173965 20537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:27.184124 20537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:27.186203 20537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:27.186316 20537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:27.190109 20537 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:41205
I20260928 12:28:27.190191 20604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:41205 every 8 connection(s)
I20260928 12:28:27.190569 20537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:27.191452 20605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:27.193086 20605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:27.200459 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20537
I20260928 12:28:27.200562 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:28:27.200798 7245 external_mini_cluster.cc:1428] Setting key 107b065b115970921fd8475dbe3aceef
I20260928 12:28:27.203121 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.193844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.7.19.126:41205)
0928 12:28:27.194116 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.194120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.194145 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:28:27.194666 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.194668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.194685 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.194768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.195303 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.196142 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.197065 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.197375 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.197474 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.199541 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.199548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.199548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.199555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.201394 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.201821 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.201824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.201824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.201934 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.202147 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.202148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.202149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.202481 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.202554 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.202781 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.202936 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260928 12:28:27.203156 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.193836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41205 (local address 127.0.0.1:53972)
0928 12:28:27.194021 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.194055 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.194502 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.194794 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.194796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.195047 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.195212 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.195216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.196222 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.196223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.196975 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.196977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.197462 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.197463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.197517 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.198075 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.198083 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.199439 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.201475 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.201481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.201483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.201741 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.202003 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.202003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.202004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.202084 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.202558 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.202564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.202652 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.202764 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.203004 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":58,"thread_start_us":105,"threads_started":1}
I20260928 12:28:27.204138 20605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:28:27.204536 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.204402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50356)
0928 12:28:27.204453 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.204504 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:28:27.204756 20554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
I20260928 12:28:27.205252 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.205143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50368)
0928 12:28:27.205192 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.205223 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:28:27.205416 20605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:34363: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.7.19.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab.
Entry for principal kudu/127.7.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab.
I20260928 12:28:27.228451 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:34363
--webserver_interface=127.7.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab
--principal=kudu/127.7.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:41205,127.7.19.125:34363,127.7.19.124:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:27.254647 20605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } attempt: 1
I20260928 12:28:27.255156 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.254999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50380)
0928 12:28:27.255092 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.255125 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 12:28:27.255520 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.255425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50390)
0928 12:28:27.255469 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.255496 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:28:27.255612 20605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:34363: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
W20260928 12:28:27.308878 20612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:27.309078 20612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:27.309101 20612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:27.310211 20612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:27.310253 20612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:27.310267 20612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:27.310280 20612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:27.310292 20612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:27.311785 20612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:41205,127.7.19.125:34363,127.7.19.124:40917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab
--principal=kudu/127.7.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:27.312026 20612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:27.312259 20612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:27.317032 20618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:27.317029 20617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:27.317165 20612 server_base.cc:1034] running on GCE node
W20260928 12:28:27.317056 20620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:27.317492 20612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:27.317797 20612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:27.318967 20612 hybrid_clock.cc:648] HybridClock initialized: now 1790598507318961 us; error 35 us; skew 500 ppm
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:27.321527 20612 init.cc:377] Logged in from keytab as kudu/127.7.19.125@KRBTEST.COM (short username kudu)
I20260928 12:28:27.321954 20612 webserver.cc:460] Webserver started at http://127.7.19.125:46031/ using document root <none> and password file <none>
I20260928 12:28:27.322198 20612 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:27.322268 20612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:27.322418 20612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:27.323480 20612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data/instance:
uuid: "3d1a3fa6acbf4459900ab8df20f905ed"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "ddb53b33750d350ee8867061da3864b3"
server_key_iv: "b7da3c9d84a78a58686bfdbc83ed4203"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.323740 20605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } attempt: 2
I20260928 12:28:27.323868 20612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal/instance:
uuid: "3d1a3fa6acbf4459900ab8df20f905ed"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "ddb53b33750d350ee8867061da3864b3"
server_key_iv: "b7da3c9d84a78a58686bfdbc83ed4203"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.324108 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.323986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50402)
0928 12:28:27.324035 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.324077 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:28:27.324438 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.324357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50410)
0928 12:28:27.324400 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.324422 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:28:27.324551 20605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:34363: Network error: Client connection negotiation failed: client connection to 127.7.19.125:34363: connect: Connection refused (error 111)
I20260928 12:28:27.325448 20612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:28:27.326162 20627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:27.326404 20612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:27.326507 20612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal
uuid: "3d1a3fa6acbf4459900ab8df20f905ed"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "ddb53b33750d350ee8867061da3864b3"
server_key_iv: "b7da3c9d84a78a58686bfdbc83ed4203"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.326607 20612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:27.335945 20612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:27.337895 20612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:27.338069 20612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:27.341936 20612 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.125:34363
I20260928 12:28:27.341967 20679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.125:34363 every 8 connection(s)
I20260928 12:28:27.342343 20612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
I20260928 12:28:27.342998 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20612
I20260928 12:28:27.343096 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-1/wal/instance
I20260928 12:28:27.343297 7245 external_mini_cluster.cc:1428] Setting key f79f11195f271f24c2ac5a4bf0124e99
I20260928 12:28:27.343425 20680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:27.345484 20680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.19.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.124@KRBTEST.COM" created.
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:27.358180 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.346220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.7.19.126:41205)
0928 12:28:27.346291 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.346294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.346543 (+ 249us) server_negotiation.cc:407] Connection header received
0928 12:28:27.347264 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.347265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.347278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.347334 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.348206 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.348948 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.350460 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.350663 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.350727 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.354999 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.355001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.355002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.355010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.356549 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.357098 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.357099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.357100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.357154 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.357481 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.357481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.357482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.357652 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.357709 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.358050 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.358116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:28:27.358497 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.346199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41205 (local address 127.0.0.1:53984)
0928 12:28:27.346401 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.346448 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.347153 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.347408 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.347412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.347800 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.348114 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.348120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.349047 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.349048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.350347 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.350350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.350764 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.350766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.350921 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.351495 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.351505 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.354906 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.356648 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.356655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.356657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.357014 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.357228 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.357229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.357229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.357385 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.357728 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.357735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.357906 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.358033 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.358212 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260928 12:28:27.359218 20680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:28:27.367411 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.359505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50422)
0928 12:28:27.359569 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.359582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.359636 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.359847 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.359848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.359995 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.360055 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.360058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.360993 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.360994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.361488 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.361489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.361986 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.361987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.362054 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.362469 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.362474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.364480 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.366318 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.366319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.366319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.366489 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.366724 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.366724 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.366725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.366781 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.367078 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.367078 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.367117 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.367206 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.367301 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 12:28:27.367491 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.359542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.7.19.125:34363)
0928 12:28:27.359693 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.359698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.359716 (+ 18us) server_negotiation.cc:407] Connection header received
0928 12:28:27.359745 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.359747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.359759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.359821 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.360135 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.360904 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.361583 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.361869 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.361973 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.364579 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.364586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.364587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.364593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.366204 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.366586 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.366589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.366590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.366654 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.366845 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.366846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.366846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.367004 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.367103 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.367237 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.367353 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260928 12:28:27.368180 20680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:28:27.368525 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.368408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54292)
0928 12:28:27.368456 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.368503 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:28:27.368786 20628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
I20260928 12:28:27.369202 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.369108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54306)
0928 12:28:27.369155 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.369183 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:28:27.369402 20680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:40917: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab.
Entry for principal kudu/127.7.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab.
I20260928 12:28:27.372733 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:40917
--webserver_interface=127.7.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab
--principal=kudu/127.7.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:36297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:41205,127.7.19.125:34363,127.7.19.124:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:27.415717 20605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } attempt: 3
I20260928 12:28:27.418680 20680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } attempt: 1
I20260928 12:28:27.419097 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.418990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54320)
0928 12:28:27.419035 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.419063 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:28:27.419409 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.419342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54332)
0928 12:28:27.419371 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.419390 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 12:28:27.419485 20680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:40917: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:28:27.423712 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.416241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.7.19.125:34363)
0928 12:28:27.416314 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.416317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.416328 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:28:27.416381 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.416383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.416393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.416475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.416831 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.417383 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.418147 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.418365 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.418462 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.420821 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.420822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.420823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.420831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.422456 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.422859 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.422860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.422860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.422923 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.423147 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.423148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.423149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.423277 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.423364 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.423532 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.423614 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:28:27.423718 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.416161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50430)
0928 12:28:27.416234 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.416245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.416317 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.416480 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.416480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.416652 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.416741 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.416744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.417523 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.417524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.418058 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.418060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.418463 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.418464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.418545 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.419055 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.419066 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.420699 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.422569 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.422570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.422570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.422763 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.422999 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.423000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.423000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.423069 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.423351 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.423351 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.423404 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.423503 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.423613 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 12:28:27.424137 20605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:28:27.424526 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.424418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54334)
0928 12:28:27.424474 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.424505 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 12:28:27.424938 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.424835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54350)
0928 12:28:27.424883 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.424909 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:28:27.425038 20605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:40917: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
W20260928 12:28:27.455014 20687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:27.455251 20687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:27.455288 20687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 12:28:27.456182 20605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } attempt: 1
W20260928 12:28:27.456501 20687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:27.456563 20687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:27.456589 20687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:27.456611 20687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:27.456632 20687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:27.456656 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.456477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54356)
0928 12:28:27.456571 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.456607 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 12:28:27.457033 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.456938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54370)
0928 12:28:27.456986 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.457010 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:28:27.457127 20605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:40917: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
I20260928 12:28:27.458325 20687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:36297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:41205,127.7.19.125:34363,127.7.19.124:40917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab
--principal=kudu/127.7.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:27.458606 20687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:27.458828 20687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:27.463740 20695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:27.463783 20692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:27.463774 20693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:27.463816 20687 server_base.cc:1034] running on GCE node
I20260928 12:28:27.464238 20687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:27.464564 20687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:27.465736 20687 hybrid_clock.cc:648] HybridClock initialized: now 1790598507465732 us; error 35 us; skew 500 ppm
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:27.468318 20687 init.cc:377] Logged in from keytab as kudu/127.7.19.124@KRBTEST.COM (short username kudu)
I20260928 12:28:27.468881 20687 webserver.cc:460] Webserver started at http://127.7.19.124:43153/ using document root <none> and password file <none>
I20260928 12:28:27.469135 20687 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:27.469206 20687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:27.469358 20687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:27.470435 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data/instance:
uuid: "effda5a8df9344a3b3fed6cd75435696"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "dd90aae9739c66dbbe850c87d876ac9f"
server_key_iv: "ba21da85c7e724fa63d2eb72d265beef"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.470800 20687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal/instance:
uuid: "effda5a8df9344a3b3fed6cd75435696"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "dd90aae9739c66dbbe850c87d876ac9f"
server_key_iv: "ba21da85c7e724fa63d2eb72d265beef"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.472138 20687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:28:27.472915 20702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:27.473083 20687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:27.473182 20687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal
uuid: "effda5a8df9344a3b3fed6cd75435696"
format_stamp: "Formatted at 2026-09-28 12:28:27 on dist-test-slave-cb7x"
server_key: "dd90aae9739c66dbbe850c87d876ac9f"
server_key_iv: "ba21da85c7e724fa63d2eb72d265beef"
server_key_version: "encryptionkey@0"
I20260928 12:28:27.473260 20687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:27.484702 20687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:27.487015 20687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:27.487180 20687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:27.487650 20680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } attempt: 2
I20260928 12:28:27.488121 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.487964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54376)
0928 12:28:27.488035 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.488080 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 12:28:27.488518 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.488401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54392)
0928 12:28:27.488456 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.488489 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 12:28:27.488649 20680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:40917: Network error: Client connection negotiation failed: client connection to 127.7.19.124:40917: connect: Connection refused (error 111)
I20260928 12:28:27.491289 20687 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.124:40917
I20260928 12:28:27.491321 20754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.124:40917 every 8 connection(s)
I20260928 12:28:27.491663 20687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
I20260928 12:28:27.492498 20755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:27.494058 20755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:28:27.497251 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20687
I20260928 12:28:27.497332 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790598357699237-7245-0/minicluster-data/master-2/wal/instance
I20260928 12:28:27.497553 7245 external_mini_cluster.cc:1428] Setting key f7ba80c359b64cf194af26adf25c86b5
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:28:27.503676 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.494746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.7.19.126:41205)
0928 12:28:27.494830 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.494833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.495011 (+ 178us) server_negotiation.cc:407] Connection header received
0928 12:28:27.495495 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.495496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.495508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.495549 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.496162 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.496657 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.497809 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.498112 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.498229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.500461 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.500462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.500463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.500469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.502006 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.502524 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.502525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.502525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.502580 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.502903 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.502921 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.502922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.503056 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.503121 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.503491 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.503583 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:28:27.503768 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.494713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41205 (local address 127.0.0.1:53998)
0928 12:28:27.494896 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.494928 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.495382 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.495638 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.495641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.495884 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.496066 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.496070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.496777 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.496778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.497696 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.497698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.498234 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.498235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.498373 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.498938 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.498946 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.500348 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.502107 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.502114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.502116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.502435 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.502658 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.502659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.502659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.502790 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.503138 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.503145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.503307 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.503411 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.503529 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260928 12:28:27.504369 20755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:28:27.506659 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.497958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.7.19.124:40917)
0928 12:28:27.498127 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.498132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.498156 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:28:27.498210 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.498212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.498233 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.498298 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.498879 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.499609 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.500395 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.500804 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.500861 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.503317 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.503324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.503324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.503329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.505479 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.505831 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.505833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.505834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.505898 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.506105 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.506106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.506106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.506251 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.506438 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.506481 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.506553 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:28:27.511065 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.504636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:34363 (local address 127.0.0.1:50444)
0928 12:28:27.504683 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.504693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.504738 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.504895 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.504895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.505050 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.505086 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.505088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.505705 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.505706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.506163 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.506165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.506528 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.506528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.506593 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.506995 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.507004 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.508283 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.510028 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.510029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.510030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.510212 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.510428 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.510428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.510429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.510490 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.510748 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.510749 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.510790 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.510878 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.510966 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 12:28:27.511061 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.504733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.7.19.125:34363)
0928 12:28:27.504778 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.504780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.504788 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:28:27.504811 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.504812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.504828 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.504874 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.505168 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.505609 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.506249 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.506453 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.506531 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.508366 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.508367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.508367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.508374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.509923 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.510293 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.510294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.510295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.510351 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.510568 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.510569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.510569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.510677 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.510755 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.510903 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.510973 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:28:27.511377 20755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:27.515417 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.507482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.7.19.126:41205)
0928 12:28:27.507520 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.507522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.507531 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:28:27.507610 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.507611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.507621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.507688 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.508125 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.508787 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.509754 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.509927 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.510035 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.512276 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.512277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.512277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.512282 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.514067 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.514492 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.514493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.514493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.514579 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.514792 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.514793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.514793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.514928 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.515137 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.515184 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.515294 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:28:27.518846 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.511651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54418)
0928 12:28:27.511710 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.511721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.511776 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.511936 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.511937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.512055 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.512078 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.512079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.512914 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.512915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.513768 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.513771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.514179 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.514180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.514233 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.514612 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.514618 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.516085 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.517754 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.517755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.517755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.517925 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.518148 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.518149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.518150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.518207 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.518486 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.518487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.518527 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.518622 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.518723 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 12:28:27.518906 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.511681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.7.19.124:40917)
0928 12:28:27.511713 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.511715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.511761 (+ 46us) server_negotiation.cc:407] Connection header received
0928 12:28:27.511830 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.511830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.511857 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.511907 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.512154 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.512827 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.513859 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.514066 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.514174 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.516194 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.516195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.516196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.516203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.517650 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.518018 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.518019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.518019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.518078 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.518284 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.518285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.518285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.518398 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.518493 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.518645 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.518779 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":16}
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:28:27.520388 20755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696: Bootstrap starting.
I20260928 12:28:27.521443 20755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:27.521763 20755 log.cc:824] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:27.522516 20755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696: No bootstrap required, opened a new log
I20260928 12:28:27.523442 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.516005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.7.19.125:34363)
0928 12:28:27.516050 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.516053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.516062 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:28:27.516082 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.516083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.516092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.516161 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.516717 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.517303 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.517960 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.518159 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.518242 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.520718 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.520720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.520720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.520728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.522188 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.522571 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.522572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.522572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.522656 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.522869 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.522870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.522871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.522996 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.523204 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.523244 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.523322 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:28:27.524114 20755 raft_consensus.cc:348] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.524255 20755 raft_consensus.cc:374] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:27.524302 20755 raft_consensus.cc:729] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: effda5a8df9344a3b3fed6cd75435696, State: Initialized, Role: FOLLOWER
I20260928 12:28:27.524456 20755 consensus_queue.cc:260] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.524739 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.524816 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:28:27.524824 20755 sys_catalog.cc:564] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:27.525287 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 12:28:27.526643 20772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:28:27.526700 20772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:28:27.527101 20687 master_runner.cc:186] Error getting master registration for 127.7.19.126:41205: OK, instance_id { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" instance_seqno: 1790598507189017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:28:27.529796 20687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:28:27.532333 20605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } attempt: 2
I20260928 12:28:27.536542 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.526963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.7.19.124:40917)
0928 12:28:27.527120 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.527122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.527577 (+ 455us) server_negotiation.cc:407] Connection header received
0928 12:28:27.527677 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.527678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.527687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.527718 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.528480 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.529272 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.530321 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.530612 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.530679 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.532275 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.532276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.532277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.532284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.534132 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.534601 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.534603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.534603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.534663 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.535159 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.535160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.535160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.535803 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.535885 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.536353 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.536446 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 12:28:27.537798 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.527310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.7.19.126:41205)
0928 12:28:27.527349 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.527352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.527573 (+ 221us) server_negotiation.cc:407] Connection header received
0928 12:28:27.527752 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.527753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.527763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.527792 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.529441 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.530138 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.531564 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.531850 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.531891 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.534501 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.534503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.534503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.534511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.536311 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.536786 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.536787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.536788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.536849 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.537077 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.537078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.537079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.537222 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.537307 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.537577 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.537669 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:28:27.539105 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.527141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.7.19.125:34363)
0928 12:28:27.527182 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.527184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.527708 (+ 524us) server_negotiation.cc:407] Connection header received
0928 12:28:27.528626 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.528627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.528638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.528694 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.530402 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.531112 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.532295 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.532649 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.532741 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.535368 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.535369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.535369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.535376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.537105 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.537517 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.537519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.537520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.537578 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.537790 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.537791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.537791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.537931 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.538699 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.538919 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.539000 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:28:27.544420 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.532833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54448)
0928 12:28:27.532930 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.532941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.532976 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.537066 (+ 4090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.537067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.537229 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.537273 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.537275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.538171 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.538172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.538967 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.538969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.539727 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.539728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.539778 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.540148 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.540152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.541601 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.543395 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.543395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.543396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.543608 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.543822 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.543822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.543823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.543877 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.544167 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.544168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.544213 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.544281 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.544353 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 12:28:27.544486 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.532931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.7.19.124:40917)
0928 12:28:27.536925 (+ 3994us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.536927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.536934 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:28:27.536952 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.536953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.536961 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.537340 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.537367 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.538071 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.539157 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.539653 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.539712 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.541680 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.541681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.541682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.541688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.543307 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.543687 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.543688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.543688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.543746 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.543940 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.543941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.543942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.544071 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.544137 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.544323 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.544395 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3981,"thread_start_us":93,"threads_started":1}
I20260928 12:28:27.546104 20605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: Bootstrap starting.
I20260928 12:28:27.546851 20605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:27.547155 20605 log.cc:824] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:27.548017 20605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: No bootstrap required, opened a new log
I20260928 12:28:27.550031 20605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.550174 20605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:27.550207 20605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfd3a2492ce4bfdb2c7a033bf2d96f7, State: Initialized, Role: FOLLOWER
I20260928 12:28:27.550350 20605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.550618 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.550661 20605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:27.550730 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:28:27.552543 20794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:28:27.552650 20794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:28:27.553191 20537 master_runner.cc:186] Error getting master registration for 127.7.19.125:34363: OK, instance_id { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" instance_seqno: 1790598507340799 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:28:27.555452 20537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:28:27.579836 20680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } attempt: 3
Sep 28 12:28:27 dist-test-slave-cb7x krb5kdc[20524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598507, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:28:27.588186 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.580134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:40917 (local address 127.0.0.1:54462)
0928 12:28:27.580183 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:27.580194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:27.580232 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:27.580379 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:27.580380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:27.580575 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:27.580630 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.580632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.581666 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.581667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.582534 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:27.582536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.583002 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.583004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:27.583074 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.583592 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:27.583597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:27.585189 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:27.587063 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.587064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.587064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.587286 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.587503 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:27.587504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:27.587505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:27.587572 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:27.587853 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:27.587855 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:27.587904 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 12:28:27.587978 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 12:28:27.588082 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 12:28:27.588187 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:27.580194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.7.19.124:40917)
0928 12:28:27.580248 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:27.580250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:27.580259 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:28:27.580283 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:27.580284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:27.580293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:27.580361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:27.580732 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.581570 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.582633 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:27.582925 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:27.582991 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:27.585279 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:27.585281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:27.585281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:27.585288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:27.586970 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.587364 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.587365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.587366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.587428 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:27.587642 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:27.587643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:27.587643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:27.587779 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:27.587850 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:27.588015 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 12:28:27.588093 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:28:27.590215 20680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: Bootstrap starting.
I20260928 12:28:27.590967 20680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:27.591284 20680 log.cc:824] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:27.592396 20680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: No bootstrap required, opened a new log
I20260928 12:28:27.594439 20680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.594592 20680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:27.594628 20680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1a3fa6acbf4459900ab8df20f905ed, State: Initialized, Role: FOLLOWER
I20260928 12:28:27.594755 20680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.595078 20680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:27.595055 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.595263 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:28:27.596925 20808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:28:27.597041 20808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:28:27.598932 20612 master_runner.cc:263] No leader master found from master 3d1a3fa6acbf4459900ab8df20f905ed
I20260928 12:28:27.598975 20612 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:28:27.650451 20761 raft_consensus.cc:482] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:27.650579 20761 raft_consensus.cc:504] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.650811 20761 leader_election.cc:288] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dfd3a2492ce4bfdb2c7a033bf2d96f7 (127.7.19.126:41205), 3d1a3fa6acbf4459900ab8df20f905ed (127.7.19.125:34363)
I20260928 12:28:27.651352 20579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "effda5a8df9344a3b3fed6cd75435696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" is_pre_election: true
I20260928 12:28:27.651520 20579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate effda5a8df9344a3b3fed6cd75435696 in term 0.
I20260928 12:28:27.651857 20705 leader_election.cc:302] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dfd3a2492ce4bfdb2c7a033bf2d96f7, effda5a8df9344a3b3fed6cd75435696; no voters:
I20260928 12:28:27.652220 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "effda5a8df9344a3b3fed6cd75435696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" is_pre_election: true
I20260928 12:28:27.652365 20654 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate effda5a8df9344a3b3fed6cd75435696 in term 0.
I20260928 12:28:27.652518 20761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:28:27.652577 20761 raft_consensus.cc:482] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:28:27.652601 20761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:27.653563 20761 raft_consensus.cc:504] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.653683 20761 leader_election.cc:288] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [CANDIDATE]: Term 1 election: Requested vote from peers 9dfd3a2492ce4bfdb2c7a033bf2d96f7 (127.7.19.126:41205), 3d1a3fa6acbf4459900ab8df20f905ed (127.7.19.125:34363)
I20260928 12:28:27.654130 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "effda5a8df9344a3b3fed6cd75435696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1a3fa6acbf4459900ab8df20f905ed"
I20260928 12:28:27.654203 20654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:27.655018 20654 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate effda5a8df9344a3b3fed6cd75435696 in term 1.
I20260928 12:28:27.655242 20704 leader_election.cc:302] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1a3fa6acbf4459900ab8df20f905ed, effda5a8df9344a3b3fed6cd75435696; no voters:
I20260928 12:28:27.655364 20579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "effda5a8df9344a3b3fed6cd75435696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7"
I20260928 12:28:27.655433 20579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:27.656176 20579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate effda5a8df9344a3b3fed6cd75435696 in term 1.
I20260928 12:28:27.657189 20761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:27.657341 20761 raft_consensus.cc:686] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [term 1 LEADER]: Becoming Leader. State: Replica: effda5a8df9344a3b3fed6cd75435696, State: Running, Role: LEADER
I20260928 12:28:27.657434 20761 consensus_queue.cc:237] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } }
I20260928 12:28:27.657752 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader effda5a8df9344a3b3fed6cd75435696. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.657806 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:27.658075 20812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:27.658270 20812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:27.660187 20654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [term 1 FOLLOWER]: Refusing update from remote peer effda5a8df9344a3b3fed6cd75435696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:27.660393 20579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [term 1 FOLLOWER]: Refusing update from remote peer effda5a8df9344a3b3fed6cd75435696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:27.660754 20761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:27.662034 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader effda5a8df9344a3b3fed6cd75435696. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.662127 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:28:27.662321 20761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:27.663439 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader effda5a8df9344a3b3fed6cd75435696. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.663514 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:28:27.665014 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.665092 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:27.665524 20797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.665594 20797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:28:27.665834 20813 mvcc.cc:205] Tried to move back new op lower bound from 7334291487373598720 to 7334291487365382144. Current Snapshot: MvccSnapshot[applied={T|T < 7334291487373598720}]
I20260928 12:28:27.666042 20816 mvcc.cc:205] Tried to move back new op lower bound from 7334291487373598720 to 7334291487365382144. Current Snapshot: MvccSnapshot[applied={T|T < 7334291487373598720}]
I20260928 12:28:27.666828 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.666899 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:27.667268 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "effda5a8df9344a3b3fed6cd75435696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dfd3a2492ce4bfdb2c7a033bf2d96f7" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41205 } } peers { permanent_uuid: "3d1a3fa6acbf4459900ab8df20f905ed" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 34363 } } peers { permanent_uuid: "effda5a8df9344a3b3fed6cd75435696" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 40917 } } }
I20260928 12:28:27.667343 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:28:27.668035 20814 mvcc.cc:205] Tried to move back new op lower bound from 7334291487373598720 to 7334291487365382144. Current Snapshot: MvccSnapshot[applied={T|T < 7334291487373598720}]
I20260928 12:28:27.669014 20812 catalog_manager.cc:1347] Generated new cluster ID: 9fa17081d33d4242a8668243c592d7ac
I20260928 12:28:27.669059 20812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:28:27.681160 20812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:28:27.681710 20812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:27.685389 20812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P effda5a8df9344a3b3fed6cd75435696: Generated new TSK 0
I20260928 12:28:27.685596 20812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:28:28.553153 20794 catalog_manager.cc:1259] Loaded cluster ID: 9fa17081d33d4242a8668243c592d7ac
I20260928 12:28:28.553231 20794 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: loading cluster ID for follower catalog manager: success
I20260928 12:28:28.553977 20794 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: acquiring CA information for follower catalog manager: success
I20260928 12:28:28.554225 20794 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9dfd3a2492ce4bfdb2c7a033bf2d96f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:28:28.597532 20808 catalog_manager.cc:1259] Loaded cluster ID: 9fa17081d33d4242a8668243c592d7ac
I20260928 12:28:28.597610 20808 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: loading cluster ID for follower catalog manager: success
I20260928 12:28:28.598489 20808 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: acquiring CA information for follower catalog manager: success
I20260928 12:28:28.598806 20808 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d1a3fa6acbf4459900ab8df20f905ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:28:28.745450 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20537
I20260928 12:28:28.750916 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20612
I20260928 12:28:28.755820 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20687
2026-09-28T12:28:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2890 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:28:28 dist-test-slave-cb7x krb5kdc[20827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:28:28 dist-test-slave-cb7x krb5kdc[20827](info): set up 2 sockets
Sep 28 12:28:28 dist-test-slave-cb7x krb5kdc[20827](info): commencing operation
krb5kdc: starting...
W20260928 12:28:29.820911 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:28:29 dist-test-slave-cb7x krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:28:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:28:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:28:29.939965 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:42155
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:30.018934 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:30.019174 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:30.019209 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:30.020326 20840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:30.020393 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:30.020417 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:30.020452 20840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:30.020478 20840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:30.021961 20840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:30.022243 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:30.022464 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:30.027154 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:30.027187 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:30.027172 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:30.027349 20840 server_base.cc:1034] running on GCE node
I20260928 12:28:30.027635 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:30.027869 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:30.029065 20840 hybrid_clock.cc:648] HybridClock initialized: now 1790598510029039 us; error 54 us; skew 500 ppm
Sep 28 12:28:30 dist-test-slave-cb7x krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598510, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:30.031533 20840 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:30.031942 20840 webserver.cc:460] Webserver started at http://127.7.19.126:44617/ using document root <none> and password file <none>
I20260928 12:28:30.032186 20840 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:30.032230 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:30.032388 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:30.033430 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "fa2bf7a218bd489884a69c7867676076"
format_stamp: "Formatted at 2026-09-28 12:28:30 on dist-test-slave-cb7x"
server_key: "d331605c8453d68e4d11419048361b24"
server_key_iv: "e9dd62d2b6a76502bafb09a24cbf5d2e"
server_key_version: "encryptionkey@0"
I20260928 12:28:30.033809 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "fa2bf7a218bd489884a69c7867676076"
format_stamp: "Formatted at 2026-09-28 12:28:30 on dist-test-slave-cb7x"
server_key: "d331605c8453d68e4d11419048361b24"
server_key_iv: "e9dd62d2b6a76502bafb09a24cbf5d2e"
server_key_version: "encryptionkey@0"
I20260928 12:28:30.035104 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:28:30.035861 20855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:30.036059 20840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:30.036144 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "fa2bf7a218bd489884a69c7867676076"
format_stamp: "Formatted at 2026-09-28 12:28:30 on dist-test-slave-cb7x"
server_key: "d331605c8453d68e4d11419048361b24"
server_key_iv: "e9dd62d2b6a76502bafb09a24cbf5d2e"
server_key_version: "encryptionkey@0"
I20260928 12:28:30.036228 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:30.054095 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:30.056151 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:30.056296 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:30.060030 20840 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:42155
I20260928 12:28:30.060107 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:42155 every 8 connection(s)
I20260928 12:28:30.060451 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:30.061342 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:30.063575 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076: Bootstrap starting.
I20260928 12:28:30.064088 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20840
I20260928 12:28:30.064185 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:28:30.064221 20908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:30.064415 7245 external_mini_cluster.cc:1428] Setting key f91b4a76ae79fca4673b6bba621c310e
I20260928 12:28:30.064574 20908 log.cc:824] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:30.065387 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076: No bootstrap required, opened a new log
I20260928 12:28:30.066963 20908 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2bf7a218bd489884a69c7867676076" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 42155 } }
I20260928 12:28:30.067126 20908 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:30.067183 20908 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2bf7a218bd489884a69c7867676076, State: Initialized, Role: FOLLOWER
I20260928 12:28:30.067313 20908 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2bf7a218bd489884a69c7867676076" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 42155 } }
I20260928 12:28:30.067417 20908 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:30.067449 20908 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:30.067497 20908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:30.068406 20908 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2bf7a218bd489884a69c7867676076" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 42155 } }
I20260928 12:28:30.068514 20908 leader_election.cc:302] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa2bf7a218bd489884a69c7867676076; no voters:
I20260928 12:28:30.068708 20908 leader_election.cc:288] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:28:30.068755 20913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:30.068908 20913 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [term 1 LEADER]: Becoming Leader. State: Replica: fa2bf7a218bd489884a69c7867676076, State: Running, Role: LEADER
I20260928 12:28:30.068948 20908 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:30.069017 20913 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2bf7a218bd489884a69c7867676076" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 42155 } }
I20260928 12:28:30.069306 20913 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa2bf7a218bd489884a69c7867676076. Latest consensus state: current_term: 1 leader_uuid: "fa2bf7a218bd489884a69c7867676076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2bf7a218bd489884a69c7867676076" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 42155 } } }
I20260928 12:28:30.069397 20913 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:30.069316 20915 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa2bf7a218bd489884a69c7867676076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2bf7a218bd489884a69c7867676076" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 42155 } } }
I20260928 12:28:30.069556 20915 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:30.069676 20920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:30.070156 20920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:30.072962 20920 catalog_manager.cc:1347] Generated new cluster ID: 6bdfd9df2e5241e19fe6e3cfe279a63e
I20260928 12:28:30.073033 20920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:28:30 dist-test-slave-cb7x krb5kdc[20827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:30.078399 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:30.064964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.7.19.126:42155)
0928 12:28:30.065150 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:30.065166 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:30.065189 (+ 23us) server_negotiation.cc:407] Connection header received
0928 12:28:30.067350 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:30.067358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:30.067416 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:30.067560 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:30.068194 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:30.069648 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:30.070732 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:30.071342 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:30.071464 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:30.074228 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:30.074232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:30.074232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:30.074244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:30.076340 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:30.076746 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:30.076749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:30.076749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:30.076864 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:30.077068 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:30.077069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:30.077069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:30.077377 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:30.077963 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:30.078035 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:28:30.078192 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 12:28:30.079303 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:28:30.093911 20920 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:28:30.094651 20920 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:30.099356 20920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa2bf7a218bd489884a69c7867676076: Generated new TSK 0
I20260928 12:28:30.099560 20920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:28:30.133293 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20840
2026-09-28T12:28:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1375 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:28:30 dist-test-slave-cb7x krb5kdc[20935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:28:30 dist-test-slave-cb7x krb5kdc[20935](info): set up 2 sockets
Sep 28 12:28:30 dist-test-slave-cb7x krb5kdc[20935](info): commencing operation
krb5kdc: starting...
W20260928 12:28:31.209201 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:28:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:28:31.332381 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:33465
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:33465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:31.411765 20948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:31.412009 20948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:31.412046 20948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:31.413213 20948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:31.413282 20948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:31.413306 20948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:31.413328 20948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:31.413362 20948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:31.414841 20948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:33465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:31.415122 20948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:31.415345 20948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:31.420258 20956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.420352 20954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.420388 20953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:31.420400 20948 server_base.cc:1034] running on GCE node
I20260928 12:28:31.420872 20948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:31.421177 20948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:31.422371 20948 hybrid_clock.cc:648] HybridClock initialized: now 1790598511422347 us; error 36 us; skew 500 ppm
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:31.424945 20948 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:31.425390 20948 webserver.cc:460] Webserver started at http://127.7.19.126:36657/ using document root <none> and password file <none>
I20260928 12:28:31.425604 20948 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:31.425647 20948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:31.425732 20948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:31.426752 20948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "f3eeef8a2d454e0ebc90c732be175bd5"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "87b7a645f9c61d94a69f00854a317809"
server_key_iv: "9614616ec0a2cb2e984439d21b3a8691"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.427052 20948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "f3eeef8a2d454e0ebc90c732be175bd5"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "87b7a645f9c61d94a69f00854a317809"
server_key_iv: "9614616ec0a2cb2e984439d21b3a8691"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.428282 20948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:28:31.428980 20963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.429155 20948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:28:31.429236 20948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "f3eeef8a2d454e0ebc90c732be175bd5"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "87b7a645f9c61d94a69f00854a317809"
server_key_iv: "9614616ec0a2cb2e984439d21b3a8691"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.429289 20948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:31.441108 20948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:31.443286 20948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:31.443396 20948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:31.447101 20948 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:33465
I20260928 12:28:31.447157 21015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:33465 every 8 connection(s)
I20260928 12:28:31.447467 20948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:31.448465 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:31.450692 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap starting.
I20260928 12:28:31.451289 21016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:31.451629 21016 log.cc:824] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:31.452360 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: No bootstrap required, opened a new log
I20260928 12:28:31.454160 21016 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:31.454310 21016 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:31.454335 21016 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3eeef8a2d454e0ebc90c732be175bd5, State: Initialized, Role: FOLLOWER
I20260928 12:28:31.454489 21016 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:31.454576 21016 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:31.454622 21016 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:31.454677 21016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:31.455539 21016 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:31.455658 21016 leader_election.cc:302] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3eeef8a2d454e0ebc90c732be175bd5; no voters:
I20260928 12:28:31.455861 21016 leader_election.cc:288] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:28:31.455971 21019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:31.456120 21019 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 LEADER]: Becoming Leader. State: Replica: f3eeef8a2d454e0ebc90c732be175bd5, State: Running, Role: LEADER
I20260928 12:28:31.456128 21016 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:31.456236 21019 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:31.456600 21020 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } } }
I20260928 12:28:31.456784 21020 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:31.456811 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 20948
I20260928 12:28:31.456931 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:28:31.457182 7245 external_mini_cluster.cc:1428] Setting key ad9d8c6fd3ec37be8cb52aaf601b5223
I20260928 12:28:31.457340 21028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:31.457897 21028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:31.458079 21021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3eeef8a2d454e0ebc90c732be175bd5. Latest consensus state: current_term: 1 leader_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } } }
I20260928 12:28:31.458256 21021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:31.459648 21028 catalog_manager.cc:1347] Generated new cluster ID: 59993feec19642a789df02a5176ed15b
I20260928 12:28:31.459705 21028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:31.470130 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.457907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.7.19.126:33465)
0928 12:28:31.458103 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:31.458113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:31.458147 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:28:31.458555 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:31.458565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:31.458631 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:31.458783 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:31.459691 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.461207 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.462358 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.463092 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.463215 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.466243 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:31.466248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:31.466248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:31.466260 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:31.468366 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.468822 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.468824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.468825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.468954 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.469183 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.469184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.469185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.469484 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:31.469642 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:31.469769 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 12:28:31.469912 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260928 12:28:31.494551 21028 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:28:31.495401 21028 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:31.513525 21028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Generated new TSK 0
I20260928 12:28:31.513748 21028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:28:31.547609 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:33465
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:31.628034 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:31.628243 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:31.628268 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:31.629470 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:31.629513 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:31.629539 21042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:28:31.630952 21042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:31.631186 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:31.631399 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:31.636430 21048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.636443 21050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.636547 21047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:31.636910 21042 server_base.cc:1034] running on GCE node
I20260928 12:28:31.637135 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:31.637408 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:31.638594 21042 hybrid_clock.cc:648] HybridClock initialized: now 1790598511638575 us; error 35 us; skew 500 ppm
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:31.641255 21042 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:28:31.641733 21042 webserver.cc:460] Webserver started at http://127.7.19.65:39461/ using document root <none> and password file <none>
I20260928 12:28:31.641956 21042 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:31.642025 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:31.642174 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:31.643198 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "f63a47250443aa51bb5ce9955484773b"
server_key_iv: "776afa63387ff87010050384a954e3bc"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.643558 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "f63a47250443aa51bb5ce9955484773b"
server_key_iv: "776afa63387ff87010050384a954e3bc"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.644886 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:28:31.645596 21057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.645766 21042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:28:31.645859 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "f63a47250443aa51bb5ce9955484773b"
server_key_iv: "776afa63387ff87010050384a954e3bc"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.645939 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:31.659597 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:31.661888 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:31.662057 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:31.662298 21042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:31.662629 21042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:31.662686 21042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.662720 21042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:31.662747 21042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.668351 21042 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:41507
I20260928 12:28:31.668422 21170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:41507 every 8 connection(s)
I20260928 12:28:31.668803 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:28:31.672415 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21042
I20260928 12:28:31.672567 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:28:31.672799 7245 external_mini_cluster.cc:1428] Setting key dc106d0f2e69807b9176c3bf7eae5d11
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:31.678650 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.669471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:47197 (local address 127.7.19.126:33465)
0928 12:28:31.669552 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:31.669554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:31.669744 (+ 190us) server_negotiation.cc:407] Connection header received
0928 12:28:31.670215 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:31.670216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:31.670230 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:31.670282 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:31.671196 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.671690 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.672752 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.673111 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.673186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.675676 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:31.675677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:31.675677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:31.675684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:31.677158 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.677678 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.677679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.677679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.677730 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.678021 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.678021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.678022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.678125 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:31.678179 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:31.678522 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 12:28:31.678587 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:28:31.678930 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.669409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:47197)
0928 12:28:31.669587 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:31.669641 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:31.670125 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:31.670333 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:31.670336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:31.670905 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:31.671093 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:31.671098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.671771 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.671772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:31.672602 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:31.672604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.673200 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.673201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:31.673341 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.673950 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:31.673957 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:31.675578 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:31.677238 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:31.677245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:31.677247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:31.677587 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:31.677797 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:31.677798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:31.677798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:31.677947 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:31.678193 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:31.678200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:31.678375 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 12:28:31.678505 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 12:28:31.678686 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260928 12:28:31.679477 21171 heartbeater.cc:347] Connected to a master server at 127.7.19.126:33465
I20260928 12:28:31.679582 21171 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:31.679776 21171 heartbeater.cc:511] Master 127.7.19.126:33465 requested a full tablet report, sending...
I20260928 12:28:31.680384 20980 ts_manager.cc:194] Registered new tserver with Master: f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507)
I20260928 12:28:31.681078 20980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:47197
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:28:31.700021 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:33465
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:31.781350 21176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:31.781597 21176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:31.781636 21176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:31.782830 21176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:31.782902 21176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:31.782946 21176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:28:31.784403 21176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:31.784682 21176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:31.784910 21176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:28:31.790215 21176 server_base.cc:1034] running on GCE node
W20260928 12:28:31.790189 21182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.790280 21181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.790280 21184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:31.790879 21176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:31.791169 21176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:31.792344 21176 hybrid_clock.cc:648] HybridClock initialized: now 1790598511792317 us; error 46 us; skew 500 ppm
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:31.794994 21176 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:28:31.795482 21176 webserver.cc:460] Webserver started at http://127.7.19.66:38477/ using document root <none> and password file <none>
I20260928 12:28:31.795719 21176 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:31.795791 21176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:31.795934 21176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:31.797068 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "1d3d8694d314401eb23e126d6eb9b17f"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "b4980f313032c067f8edaf2e9dc54975"
server_key_iv: "fcadc03fcb7955a4bed5d9ce4eb6d77e"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.797439 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "1d3d8694d314401eb23e126d6eb9b17f"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "b4980f313032c067f8edaf2e9dc54975"
server_key_iv: "fcadc03fcb7955a4bed5d9ce4eb6d77e"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.798758 21176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:28:31.799484 21191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.799654 21176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.799749 21176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "1d3d8694d314401eb23e126d6eb9b17f"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "b4980f313032c067f8edaf2e9dc54975"
server_key_iv: "fcadc03fcb7955a4bed5d9ce4eb6d77e"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.799829 21176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:31.813055 21176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:31.814903 21176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:31.815073 21176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:31.815302 21176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:31.815611 21176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:31.815665 21176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.815706 21176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:31.815734 21176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.821609 21176 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:39887
I20260928 12:28:31.821655 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:39887 every 8 connection(s)
I20260928 12:28:31.821959 21176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:28:31.824942 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21176
I20260928 12:28:31.825057 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:28:31.825251 7245 external_mini_cluster.cc:1428] Setting key 9eb2251b1a18ea4dd2c78504b7ef635f
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:31.832572 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.822632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:58199 (local address 127.7.19.126:33465)
0928 12:28:31.822701 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:31.822703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:31.822937 (+ 234us) server_negotiation.cc:407] Connection header received
0928 12:28:31.823437 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:31.823439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:31.823451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:31.823528 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:31.824654 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.825453 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.826956 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.827171 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.827234 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.829742 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:31.829743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:31.829744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:31.829754 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:31.831150 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.831592 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.831593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.831594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.831656 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.831931 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.831932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.831932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.832048 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:31.832124 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:31.832391 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 12:28:31.832482 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:28:31.832815 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.822595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:58199)
0928 12:28:31.822772 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:31.822813 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:31.823338 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:31.823728 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:31.823731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:31.824315 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:31.824557 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:31.824563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.825616 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.825617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:31.826869 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:31.826871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.827257 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.827258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:31.827404 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.827987 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:31.828002 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:31.829628 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:31.831263 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:31.831270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:31.831272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:31.831511 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:31.831749 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:31.831750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:31.831750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:31.831856 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:31.832116 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:31.832121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:31.832262 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 12:28:31.832372 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:28:31.832528 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260928 12:28:31.833323 21305 heartbeater.cc:347] Connected to a master server at 127.7.19.126:33465
I20260928 12:28:31.833411 21305 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:31.833660 21305 heartbeater.cc:511] Master 127.7.19.126:33465 requested a full tablet report, sending...
I20260928 12:28:31.834105 20980 ts_manager.cc:194] Registered new tserver with Master: 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887)
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:28:31.834604 20980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:58199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:28:31.852831 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:33465
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:31.938303 21310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:31.938527 21310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:31.938553 21310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:31.939810 21310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:31.939855 21310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:31.939913 21310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:28:31.941489 21310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:31.941756 21310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:31.942001 21310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:31.947374 21316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:31.947430 21310 server_base.cc:1034] running on GCE node
W20260928 12:28:31.947417 21315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:31.947382 21318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:31.947783 21310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:31.948056 21310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:31.949227 21310 hybrid_clock.cc:648] HybridClock initialized: now 1790598511949211 us; error 37 us; skew 500 ppm
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:31.951655 21310 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:28:31.952086 21310 webserver.cc:460] Webserver started at http://127.7.19.67:43837/ using document root <none> and password file <none>
I20260928 12:28:31.952386 21310 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:31.952442 21310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:31.952580 21310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:31.953646 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "a9decdf09ecd489eb277c890b7a1068a"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "3a815354654c78642f39e734aeb37366"
server_key_iv: "f75219f6b9cb6391777327372d8b2819"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.953997 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "a9decdf09ecd489eb277c890b7a1068a"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "3a815354654c78642f39e734aeb37366"
server_key_iv: "f75219f6b9cb6391777327372d8b2819"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.955262 21310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 12:28:31.956055 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.956256 21310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:28:31.956337 21310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "a9decdf09ecd489eb277c890b7a1068a"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "3a815354654c78642f39e734aeb37366"
server_key_iv: "f75219f6b9cb6391777327372d8b2819"
server_key_version: "encryptionkey@0"
I20260928 12:28:31.956389 21310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:31.979728 21310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:31.982053 21310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:31.982165 21310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:31.982343 21310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:31.982661 21310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:31.982697 21310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.982719 21310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:31.982765 21310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:31.988497 21310 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:38349
I20260928 12:28:31.988552 21438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:38349 every 8 connection(s)
I20260928 12:28:31.988883 21310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:28:31 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598511, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:31.997609 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.989641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:46989 (local address 127.7.19.126:33465)
0928 12:28:31.989703 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:31.989705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:31.989880 (+ 175us) server_negotiation.cc:407] Connection header received
0928 12:28:31.990356 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:31.990357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:31.990367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:31.990419 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:31.991317 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.991807 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.992849 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.993037 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.993111 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.995194 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:31.995195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:31.995195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:31.995201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:31.996371 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.996745 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.996746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.996746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.996797 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:31.997029 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:31.997030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:31.997031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:31.997146 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:31.997213 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:31.997477 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 12:28:31.997541 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 12:28:31.997677 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:31.989606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:46989)
0928 12:28:31.989781 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:31.989816 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:31.990273 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:31.990470 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:31.990473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:31.991052 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:31.991237 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:31.991242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.991890 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.991890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:31.992773 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:31.992776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:31.993142 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:31.993143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:31.993250 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:31.993780 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:31.993788 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:31.995112 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:31.996439 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:31.996444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:31.996446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:31.996645 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:31.996852 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:31.996853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:31.996853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:31.996944 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:31.997212 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:31.997215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:31.997318 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 12:28:31.997409 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 12:28:31.997517 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260928 12:28:31.997933 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21310
I20260928 12:28:31.998009 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:28:31.998204 21439 heartbeater.cc:347] Connected to a master server at 127.7.19.126:33465
I20260928 12:28:31.998221 7245 external_mini_cluster.cc:1428] Setting key 10ab797e4f66524e0513cd1e8499594c
I20260928 12:28:31.998291 21439 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:31.998467 21439 heartbeater.cc:511] Master 127.7.19.126:33465 requested a full tablet report, sending...
I20260928 12:28:31.998853 20980 ts_manager.cc:194] Registered new tserver with Master: a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
I20260928 12:28:31.999197 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:28:31.999267 20980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:46989
I20260928 12:28:32.006641 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.000517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.7.19.126:33465)
0928 12:28:32.000586 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.000588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.000662 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:28:32.000771 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.000772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.000779 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.000824 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.001327 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.001813 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.002508 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.002721 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.002804 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.004044 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.004045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.004046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.004053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.005481 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.005846 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.005847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.005848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.005902 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.006108 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.006109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.006109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.006256 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.006361 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.006472 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.006543 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:28:32.008332 20980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:28:32.010329 20980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:28:32.016685 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.012387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:38349 (local address 127.0.0.1:40978)
0928 12:28:32.012909 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.012919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.012977 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.013173 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.013177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.013180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:32.013268 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.013271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.014693 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.014694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.015621 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.015624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.016339 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.016340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.016428 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.016450 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.016530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.016582 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":9,"thread_start_us":65,"threads_started":1}
I20260928 12:28:32.016664 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.012727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.7.19.67:38349)
0928 12:28:32.012916 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.012923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.012987 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:28:32.013029 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.013031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.013046 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.013349 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:32.013418 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.014619 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.015708 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.016247 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.016338 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.016398 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.016545 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.016582 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260928 12:28:32.016830 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.012387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:41507 (local address 127.0.0.1:35918)
0928 12:28:32.012548 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.012588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.012745 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.013173 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.013177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.013182 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:32.013268 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.013271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.014637 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.014638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.015754 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.015756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.016618 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.016619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.016703 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.016706 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.016734 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.016762 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":5,"thread_start_us":57,"threads_started":1}
I20260928 12:28:32.016948 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.012746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.7.19.66:39887)
0928 12:28:32.012940 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.012945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.013161 (+ 216us) server_negotiation.cc:407] Connection header received
0928 12:28:32.013208 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.013210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.013225 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.013295 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:32.013495 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.014179 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.015609 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.016363 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.016435 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.016489 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.016805 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.016856 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260928 12:28:32.017086 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.012578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39887 (local address 127.0.0.1:59962)
0928 12:28:32.013044 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.013052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.013093 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.013372 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.013373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.013374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:32.013417 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.013418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.014290 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.014291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.015512 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.015514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.016527 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.016528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.016612 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.016617 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.016663 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.016728 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":46,"threads_started":1}
I20260928 12:28:32.017175 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.012503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.7.19.65:41507)
0928 12:28:32.012968 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.012973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.012982 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:28:32.013026 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.013028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.013046 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.013349 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:32.013417 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.014510 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.015833 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.016532 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.016938 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.017010 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.017038 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.017091 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":146,"threads_started":1}
I20260928 12:28:32.017884 21239 tablet_service.cc:1467] Processing CreateTablet for tablet 64681a557d124c5280b22698cfc9669e (DEFAULT_TABLE table=test-table [id=412fbfa63f0445ee969a9f3782bc853e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:32.018098 21373 tablet_service.cc:1467] Processing CreateTablet for tablet 64681a557d124c5280b22698cfc9669e (DEFAULT_TABLE table=test-table [id=412fbfa63f0445ee969a9f3782bc853e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:32.018211 21239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64681a557d124c5280b22698cfc9669e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:32.018416 21373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64681a557d124c5280b22698cfc9669e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:32.020766 21453 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Bootstrap starting.
I20260928 12:28:32.020918 21105 tablet_service.cc:1467] Processing CreateTablet for tablet 64681a557d124c5280b22698cfc9669e (DEFAULT_TABLE table=test-table [id=412fbfa63f0445ee969a9f3782bc853e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:32.021087 21454 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Bootstrap starting.
I20260928 12:28:32.021246 21105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64681a557d124c5280b22698cfc9669e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:32.021492 21453 tablet_bootstrap.cc:654] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:32.021705 21454 tablet_bootstrap.cc:654] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:32.021917 21453 log.cc:824] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.022254 21454 log.cc:824] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.022627 21453 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: No bootstrap required, opened a new log
I20260928 12:28:32.022691 21453 ts_tablet_manager.cc:1397] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 12:28:32.023029 21454 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: No bootstrap required, opened a new log
I20260928 12:28:32.023092 21454 ts_tablet_manager.cc:1397] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:32.024135 21453 raft_consensus.cc:348] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.024442 21453 raft_consensus.cc:374] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:32.024489 21453 raft_consensus.cc:729] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3d8694d314401eb23e126d6eb9b17f, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.024646 21453 consensus_queue.cc:260] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.024991 21453 ts_tablet_manager.cc:1428] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 12:28:32.025110 21305 heartbeater.cc:503] Master 127.7.19.126:33465 was elected leader, sending a full tablet report...
I20260928 12:28:32.025185 21457 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap starting.
I20260928 12:28:32.025354 21454 raft_consensus.cc:348] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.025818 21457 tablet_bootstrap.cc:654] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:32.025861 21454 raft_consensus.cc:374] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:32.025911 21454 raft_consensus.cc:729] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9decdf09ecd489eb277c890b7a1068a, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.026044 21454 consensus_queue.cc:260] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.026355 21454 ts_tablet_manager.cc:1428] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:28:32.026363 21439 heartbeater.cc:503] Master 127.7.19.126:33465 was elected leader, sending a full tablet report...
I20260928 12:28:32.026608 21457 log.cc:824] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.027313 21457 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: No bootstrap required, opened a new log
I20260928 12:28:32.027359 21457 ts_tablet_manager.cc:1397] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:28:32.028600 21457 raft_consensus.cc:348] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.028831 21457 raft_consensus.cc:374] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:32.028898 21457 raft_consensus.cc:729] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ddcabc1a6a4c1f91973d45b7f6af26, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.029016 21457 consensus_queue.cc:260] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.029232 21457 ts_tablet_manager.cc:1428] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:28:32.029290 21171 heartbeater.cc:503] Master 127.7.19.126:33465 was elected leader, sending a full tablet report...
W20260928 12:28:32.072489 21306 tablet.cc:2369] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:28:32.125671 21459 raft_consensus.cc:482] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:32.125813 21459 raft_consensus.cc:504] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.126327 21459 leader_election.cc:288] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507), 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887)
I20260928 12:28:32.129360 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.126619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:55631 (local address 127.7.19.65:41507)
0928 12:28:32.126671 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.126674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.126685 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:28:32.126718 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.126719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.126731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.126789 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:32.126968 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.127650 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.128536 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.129010 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.129093 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.129104 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.129252 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.129286 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 12:28:32.129374 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.126493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:41507 (local address 127.7.19.67:55631)
0928 12:28:32.126554 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.126566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.126655 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.126812 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.126813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.126815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:32.126889 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.126891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.127727 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.127728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.128443 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.128445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.129091 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.129092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.129175 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.129177 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.129246 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.129287 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 12:28:32.129390 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.126700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:54795 (local address 127.7.19.66:39887)
0928 12:28:32.126750 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.126752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.126883 (+ 131us) server_negotiation.cc:407] Connection header received
0928 12:28:32.126955 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.126956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.126968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.127017 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:32.127142 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.127696 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.128634 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.129039 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.129129 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.129139 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.129284 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.129319 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:28:32.129617 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.126613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39887 (local address 127.7.19.67:54795)
0928 12:28:32.126813 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.126821 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.126889 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.127028 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.127029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.127031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:32.127083 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.127085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.127784 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.127785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.128528 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.128530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.129130 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.129131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.129208 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.129211 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.129279 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.129318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":143,"threads_started":1}
I20260928 12:28:32.129771 21125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64681a557d124c5280b22698cfc9669e" candidate_uuid: "a9decdf09ecd489eb277c890b7a1068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" is_pre_election: true
I20260928 12:28:32.129946 21125 raft_consensus.cc:2454] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9decdf09ecd489eb277c890b7a1068a in term 0.
I20260928 12:28:32.130162 21329 leader_election.cc:302] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9decdf09ecd489eb277c890b7a1068a, f8ddcabc1a6a4c1f91973d45b7f6af26; no voters:
I20260928 12:28:32.130319 21459 raft_consensus.cc:2781] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:28:32.130406 21459 raft_consensus.cc:482] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:28:32.130432 21459 raft_consensus.cc:3037] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:32.130541 21259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64681a557d124c5280b22698cfc9669e" candidate_uuid: "a9decdf09ecd489eb277c890b7a1068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3d8694d314401eb23e126d6eb9b17f" is_pre_election: true
I20260928 12:28:32.130682 21259 raft_consensus.cc:2454] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9decdf09ecd489eb277c890b7a1068a in term 0.
I20260928 12:28:32.131247 21459 raft_consensus.cc:504] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.131346 21459 leader_election.cc:288] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [CANDIDATE]: Term 1 election: Requested vote from peers f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507), 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887)
I20260928 12:28:32.131547 21125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64681a557d124c5280b22698cfc9669e" candidate_uuid: "a9decdf09ecd489eb277c890b7a1068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
I20260928 12:28:32.131588 21259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64681a557d124c5280b22698cfc9669e" candidate_uuid: "a9decdf09ecd489eb277c890b7a1068a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3d8694d314401eb23e126d6eb9b17f"
I20260928 12:28:32.131661 21259 raft_consensus.cc:3037] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:32.131655 21125 raft_consensus.cc:3037] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:32.132575 21125 raft_consensus.cc:2454] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9decdf09ecd489eb277c890b7a1068a in term 1.
I20260928 12:28:32.132575 21259 raft_consensus.cc:2454] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9decdf09ecd489eb277c890b7a1068a in term 1.
I20260928 12:28:32.132755 21329 leader_election.cc:302] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9decdf09ecd489eb277c890b7a1068a, f8ddcabc1a6a4c1f91973d45b7f6af26; no voters:
I20260928 12:28:32.132858 21459 raft_consensus.cc:2781] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:32.133035 21459 raft_consensus.cc:686] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 LEADER]: Becoming Leader. State: Replica: a9decdf09ecd489eb277c890b7a1068a, State: Running, Role: LEADER
I20260928 12:28:32.133155 21459 consensus_queue.cc:237] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.133955 20980 catalog_manager.cc:5495] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a reported cstate change: term changed from 0 to 1, leader changed from <none> to a9decdf09ecd489eb277c890b7a1068a (127.7.19.67). New cstate: current_term: 1 leader_uuid: "a9decdf09ecd489eb277c890b7a1068a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } health_report { overall_health: UNKNOWN } } }
I20260928 12:28:32.160548 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.158166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.7.19.67:38349)
0928 12:28:32.158234 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.158236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.158244 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:28:32.158276 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.158276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.158294 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.158360 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:28:32.158487 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.158944 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.159706 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.159912 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.159991 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.160158 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:28:32.160300 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:28:32.160400 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.160425 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.160445 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:28:32.162395 21259 raft_consensus.cc:1261] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Refusing update from remote peer a9decdf09ecd489eb277c890b7a1068a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:32.162406 21125 raft_consensus.cc:1261] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Refusing update from remote peer a9decdf09ecd489eb277c890b7a1068a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:32.162842 21459 consensus_queue.cc:1035] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:32.162995 21463 consensus_queue.cc:1035] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:32.165326 21466 mvcc.cc:205] Tried to move back new op lower bound from 7334291505814364160 to 7334291505698119680. Current Snapshot: MvccSnapshot[applied={T|T < 7334291505814364160}]
I20260928 12:28:32.166719 21468 mvcc.cc:205] Tried to move back new op lower bound from 7334291505814364160 to 7334291505698119680. Current Snapshot: MvccSnapshot[applied={T|T < 7334291505814364160 or (T in {7334291505814364160})}]
I20260928 12:28:32.167171 21467 mvcc.cc:205] Tried to move back new op lower bound from 7334291505814364160 to 7334291505698119680. Current Snapshot: MvccSnapshot[applied={T|T < 7334291505814364160}]
I20260928 12:28:32.167817 20980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:32.167914 20980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:32.168707 20980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Sending DeleteTablet for 3 replicas of tablet 64681a557d124c5280b22698cfc9669e
I20260928 12:28:32.169003 21373 tablet_service.cc:1514] Processing DeleteTablet for tablet 64681a557d124c5280b22698cfc9669e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:40978
I20260928 12:28:32.169055 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21042
I20260928 12:28:32.169113 21239 tablet_service.cc:1514] Processing DeleteTablet for tablet 64681a557d124c5280b22698cfc9669e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:59962
I20260928 12:28:32.170265 21476 tablet_replica.cc:330] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: stopping tablet replica
I20260928 12:28:32.170365 21476 raft_consensus.cc:2229] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:32.170468 21476 raft_consensus.cc:2258] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:32.170969 21476 ts_tablet_manager.cc:1905] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:32.172937 21476 ts_tablet_manager.cc:1918] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:32.173019 21476 log.cc:1196] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/64681a557d124c5280b22698cfc9669e
I20260928 12:28:32.173290 21476 ts_tablet_manager.cc:1939] T 64681a557d124c5280b22698cfc9669e P 1d3d8694d314401eb23e126d6eb9b17f: Deleting consensus metadata
I20260928 12:28:32.173851 20967 catalog_manager.cc:4841] TS 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): tablet 64681a557d124c5280b22698cfc9669e (table test-table [id=412fbfa63f0445ee969a9f3782bc853e]) successfully deleted
W20260928 12:28:32.175439 20967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:41507 (error 108)
I20260928 12:28:32.175531 21475 tablet_replica.cc:330] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: stopping tablet replica
I20260928 12:28:32.175645 21475 raft_consensus.cc:2229] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:28:32.175747 21475 raft_consensus.cc:2258] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:32.175962 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21176
I20260928 12:28:32.175982 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.175843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:41507 (local address 127.0.0.1:35928)
0928 12:28:32.175899 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.175949 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:28:32.176090 20967 catalog_manager.cc:4584] TS f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64681a557d124c5280b22698cfc9669e: Network error: Client connection negotiation failed: client connection to 127.7.19.65:41507: connect: Connection refused (error 111)
I20260928 12:28:32.180609 21475 ts_tablet_manager.cc:1905] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:32.181020 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21310
I20260928 12:28:32.185012 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 20948
I20260928 12:28:32.185055 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.184925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:38349 (local address 127.0.0.1:40996)
0928 12:28:32.184985 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.185019 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.67:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:28:32.189801 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:33465
--webserver_interface=127.7.19.126
--webserver_port=36657
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:33465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:32.269682 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:32.269896 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:32.269919 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:32.271101 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:32.271147 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:32.271160 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:32.271173 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:32.271214 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:32.272711 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:33465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=36657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:32.272974 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:32.273226 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:32.281881 21487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.281919 21484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.281881 21485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:32.282061 21478 server_base.cc:1034] running on GCE node
I20260928 12:28:32.282353 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:32.282675 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:32.283836 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790598512283833 us; error 39 us; skew 500 ppm
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:32.286432 21478 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:32.286825 21478 webserver.cc:460] Webserver started at http://127.7.19.126:36657/ using document root <none> and password file <none>
I20260928 12:28:32.287029 21478 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:32.287079 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:32.288399 21478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:32.289088 21494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.289259 21478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.289350 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "f3eeef8a2d454e0ebc90c732be175bd5"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "87b7a645f9c61d94a69f00854a317809"
server_key_iv: "9614616ec0a2cb2e984439d21b3a8691"
server_key_version: "encryptionkey@0"
I20260928 12:28:32.289736 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:32.302462 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:32.304656 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:32.304823 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:32.308441 21478 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:33465
I20260928 12:28:32.308512 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:33465 every 8 connection(s)
I20260928 12:28:32.308899 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:32.310451 21547 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:28:32.311429 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap starting.
I20260928 12:28:32.314280 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21478
I20260928 12:28:32.314538 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:41507
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:33465
--webserver_port=39461
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:32.320091 21547 log.cc:824] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.321985 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:28:32.322297 21547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap complete.
I20260928 12:28:32.324503 21547 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:32.324775 21547 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3eeef8a2d454e0ebc90c732be175bd5, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.324946 21547 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:32.325001 21547 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:32.325066 21547 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:32.325126 21547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:28:32.326092 21547 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:32.326180 21547 leader_election.cc:302] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3eeef8a2d454e0ebc90c732be175bd5; no voters:
I20260928 12:28:32.326380 21547 leader_election.cc:288] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:28:32.326421 21551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:28:32.326536 21551 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 LEADER]: Becoming Leader. State: Replica: f3eeef8a2d454e0ebc90c732be175bd5, State: Running, Role: LEADER
I20260928 12:28:32.326632 21547 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:32.326632 21551 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:32.326930 21553 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } } }
I20260928 12:28:32.326933 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3eeef8a2d454e0ebc90c732be175bd5. Latest consensus state: current_term: 2 leader_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } } }
I20260928 12:28:32.327108 21553 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:32.327129 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:32.327493 21560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:32.328293 21560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:32.328433 21560 catalog_manager.cc:1259] Loaded cluster ID: 59993feec19642a789df02a5176ed15b
I20260928 12:28:32.328657 21560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:28:32.328927 21560 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 12:28:32.333989 21560 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 12:28:32.336344 21560 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:32.336526 21560 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Loaded TSK: 0
I20260928 12:28:32.336766 21560 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:28:32.398867 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:32.399209 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:32.399255 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:32.400516 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:32.400585 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:32.400632 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:28:32.402127 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:41507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=39461
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:32.402402 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:32.402647 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:32.408591 21575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.408550 21577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.408574 21574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:32.408972 21549 server_base.cc:1034] running on GCE node
I20260928 12:28:32.409195 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:32.409457 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:32.410617 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790598512410599 us; error 34 us; skew 500 ppm
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:32.413384 21549 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:28:32.413937 21549 webserver.cc:460] Webserver started at http://127.7.19.65:39461/ using document root <none> and password file <none>
I20260928 12:28:32.414202 21549 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:32.414286 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:32.416039 21549 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:28:32.416858 21584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.417140 21549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:32.417235 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "f63a47250443aa51bb5ce9955484773b"
server_key_iv: "776afa63387ff87010050384a954e3bc"
server_key_version: "encryptionkey@0"
I20260928 12:28:32.417690 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:32.443845 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:32.446324 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:32.446477 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:32.446734 21549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:32.447243 21591 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:28:32.448709 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:28:32.448764 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:28:32.448797 21549 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:28:32.449594 21549 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:28:32.449633 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:32.449805 21591 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap starting.
I20260928 12:28:32.457104 21549 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:41507
I20260928 12:28:32.457267 21698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:41507 every 8 connection(s)
I20260928 12:28:32.457543 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:28:32.460004 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21549
I20260928 12:28:32.460330 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:39887
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:33465
--webserver_port=38477
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:32.460449 21591 log.cc:824] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.463152 21591 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:28:32.463446 21591 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap complete.
I20260928 12:28:32.464372 21591 ts_tablet_manager.cc:1397] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260928 12:28:32.465615 21591 raft_consensus.cc:348] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.465890 21591 raft_consensus.cc:729] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ddcabc1a6a4c1f91973d45b7f6af26, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.466053 21591 consensus_queue.cc:260] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.466325 21591 ts_tablet_manager.cc:1428] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:32.472440 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.458517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:49879 (local address 127.7.19.126:33465)
0928 12:28:32.459563 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.459572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.459608 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:28:32.459736 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.459743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.459796 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.459917 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.460935 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.462351 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.463972 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.464556 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.464656 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.467453 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.467457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.467457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.467468 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.470621 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.471075 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.471078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.471079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.471183 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.471482 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.471483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.471483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.471739 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.471835 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.472121 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.472247 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":88,"threads_started":1}
I20260928 12:28:32.472457 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.458193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:49879)
0928 12:28:32.458706 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.458758 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.459361 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.460044 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.460047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.460420 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:32.460743 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.460750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.462514 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.462515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.463871 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.463874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.464790 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.464791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.464930 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.465542 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:32.465554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:32.467343 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:32.470738 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.470744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.470747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.470970 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.471266 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.471267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.471268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.471407 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.471816 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:32.471822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:32.471957 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.472067 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.472247 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":80,"threads_started":1}
I20260928 12:28:32.473239 21699 heartbeater.cc:347] Connected to a master server at 127.7.19.126:33465
I20260928 12:28:32.473332 21699 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:32.473508 21699 heartbeater.cc:511] Master 127.7.19.126:33465 requested a full tablet report, sending...
I20260928 12:28:32.474196 21511 ts_manager.cc:194] Registered new tserver with Master: f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507)
I20260928 12:28:32.475292 21511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:49879
I20260928 12:28:32.476243 21699 heartbeater.cc:503] Master 127.7.19.126:33465 was elected leader, sending a full tablet report...
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.65@KRBTEST.COM
I20260928 12:28:32.483661 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.475001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.7.19.65:41507)
0928 12:28:32.475606 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.475611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.475635 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:28:32.475676 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.475678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.475694 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.475799 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.476328 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.477583 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.478331 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.478646 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.478723 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.480867 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.480873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.480873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.480879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.482437 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.482862 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.482864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.482864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.482916 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.483127 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.483128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.483128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.483246 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.483307 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.483501 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.483569 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":67,"threads_started":1}
I20260928 12:28:32.483696 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.474771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:41507 (local address 127.0.0.1:55348)
0928 12:28:32.475360 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.475400 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.475472 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.475808 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.475813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.476164 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:32.476219 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.476221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.477676 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.477677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.478225 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.478226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.478733 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.478735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.478810 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.479297 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:32.479302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:32.480791 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:32.482521 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.482527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.482528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.482784 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.482984 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.482985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.482986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.483060 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.483317 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:32.483324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:32.483379 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.483473 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.483586 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":87,"threads_started":1}
I20260928 12:28:32.484077 21633 tablet_service.cc:1514] Processing DeleteTablet for tablet 64681a557d124c5280b22698cfc9669e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 12:28:32.484635 21708 tablet_replica.cc:330] stopping tablet replica
I20260928 12:28:32.484762 21708 raft_consensus.cc:2229] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:32.484830 21708 raft_consensus.cc:2258] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:32.485133 21708 ts_tablet_manager.cc:1905] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:32.487475 21708 ts_tablet_manager.cc:1918] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:32.487551 21708 log.cc:1196] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/64681a557d124c5280b22698cfc9669e
I20260928 12:28:32.487789 21708 ts_tablet_manager.cc:1939] T 64681a557d124c5280b22698cfc9669e P f8ddcabc1a6a4c1f91973d45b7f6af26: Deleting consensus metadata
I20260928 12:28:32.488270 21498 catalog_manager.cc:4841] TS f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507): tablet 64681a557d124c5280b22698cfc9669e (table test-table [id=412fbfa63f0445ee969a9f3782bc853e]) successfully deleted
W20260928 12:28:32.551172 21704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:32.551360 21704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:32.551383 21704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:32.552425 21704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:32.552459 21704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:32.552484 21704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:28:32.553891 21704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:39887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=38477
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:32.554064 21704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:32.554229 21704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:32.559043 21714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.559065 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.559109 21716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:32.559230 21704 server_base.cc:1034] running on GCE node
I20260928 12:28:32.559422 21704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:32.559680 21704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:32.560840 21704 hybrid_clock.cc:648] HybridClock initialized: now 1790598512560817 us; error 36 us; skew 500 ppm
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:32.563195 21704 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:28:32.563633 21704 webserver.cc:460] Webserver started at http://127.7.19.66:38477/ using document root <none> and password file <none>
I20260928 12:28:32.563858 21704 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:32.563913 21704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:32.565265 21704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:32.565837 21723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.566008 21704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:28:32.566102 21704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "1d3d8694d314401eb23e126d6eb9b17f"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "b4980f313032c067f8edaf2e9dc54975"
server_key_iv: "fcadc03fcb7955a4bed5d9ce4eb6d77e"
server_key_version: "encryptionkey@0"
I20260928 12:28:32.566421 21704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:32.582453 21704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:32.584329 21704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:32.584491 21704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:32.584801 21704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:32.585114 21704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:32.585177 21704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.585217 21704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:32.585245 21704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.591055 21704 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:39887
I20260928 12:28:32.591105 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:39887 every 8 connection(s)
I20260928 12:28:32.591439 21704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:32.600807 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.592180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:45849 (local address 127.7.19.126:33465)
0928 12:28:32.592227 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.592229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.592407 (+ 178us) server_negotiation.cc:407] Connection header received
0928 12:28:32.593278 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.593278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.593288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.593332 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.594285 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.594790 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.595804 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.595999 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.596051 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.598136 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.598137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.598137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.598143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.599551 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.599955 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.599956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.599956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.600008 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.600212 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.600213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.600213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.600309 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.600359 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.600588 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.600655 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:28:32.600929 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.592169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:45849)
0928 12:28:32.592305 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.592338 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.593185 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.593403 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.593407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.594003 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:32.594191 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.594196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.594894 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.594895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.595719 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.595721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.596086 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.596087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.596192 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.596746 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:32.596762 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:32.598057 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:32.599680 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.599687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.599690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.599893 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.600068 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.600069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.600069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.600159 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.600363 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:32.600367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:32.600473 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.600548 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.600729 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260928 12:28:32.601455 21837 heartbeater.cc:347] Connected to a master server at 127.7.19.126:33465
I20260928 12:28:32.601567 21837 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:32.601634 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21704
I20260928 12:28:32.601764 21837 heartbeater.cc:511] Master 127.7.19.126:33465 requested a full tablet report, sending...
I20260928 12:28:32.601917 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:38349
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:33465
--webserver_port=43837
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:32.602217 21511 ts_manager.cc:194] Registered new tserver with Master: 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887)
I20260928 12:28:32.602720 21511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:45849
W20260928 12:28:32.682518 21840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:32.682710 21840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:32.682735 21840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:32.683799 21840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:32.683831 21840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:32.683856 21840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:28:32.685235 21840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:38349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=43837
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:32.685413 21840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:32.685578 21840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:32.690378 21845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.690379 21846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:32.690397 21848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:32.690706 21840 server_base.cc:1034] running on GCE node
I20260928 12:28:32.690897 21840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:32.691084 21840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:32.692245 21840 hybrid_clock.cc:648] HybridClock initialized: now 1790598512692219 us; error 39 us; skew 500 ppm
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:32.694751 21840 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:28:32.695106 21840 webserver.cc:460] Webserver started at http://127.7.19.67:43837/ using document root <none> and password file <none>
I20260928 12:28:32.695287 21840 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:32.695338 21840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:32.696727 21840 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:32.697516 21855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:32.697705 21840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:28:32.697777 21840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "a9decdf09ecd489eb277c890b7a1068a"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "3a815354654c78642f39e734aeb37366"
server_key_iv: "f75219f6b9cb6391777327372d8b2819"
server_key_version: "encryptionkey@0"
I20260928 12:28:32.698144 21840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:32.725102 21840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:32.727588 21840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:32.727744 21840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:32.727983 21840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:32.728461 21862 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:28:32.729916 21840 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:28:32.729993 21840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:28:32.730039 21840 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:28:32.730664 21840 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:28:32.730717 21840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:32.730780 21862 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Bootstrap starting.
I20260928 12:28:32.732251 21862 log.cc:824] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.734485 21862 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:28:32.734782 21862 tablet_bootstrap.cc:492] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Bootstrap complete.
I20260928 12:28:32.734952 21862 ts_tablet_manager.cc:1397] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 12:28:32.736255 21862 raft_consensus.cc:348] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.736552 21862 raft_consensus.cc:729] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9decdf09ecd489eb277c890b7a1068a, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.736763 21862 consensus_queue.cc:260] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } } peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } }
I20260928 12:28:32.737150 21862 ts_tablet_manager.cc:1428] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 12:28:32.737764 21840 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:38349
I20260928 12:28:32.737831 21971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:38349 every 8 connection(s)
I20260928 12:28:32.738119 21840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:32.747000 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.738817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:45855 (local address 127.7.19.126:33465)
0928 12:28:32.738910 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.738912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.739069 (+ 157us) server_negotiation.cc:407] Connection header received
0928 12:28:32.739253 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.739254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.739266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.739316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.739927 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.740440 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.741523 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.741745 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.741834 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.744380 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.744382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.744383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.744390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.745611 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.746021 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.746022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.746022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.746084 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.746354 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.746355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.746355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.746469 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.746539 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.746823 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.746908 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 12:28:32.747211 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.738775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:45855)
0928 12:28:32.738941 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.738988 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.739157 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.739368 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.739372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.739645 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:32.739830 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.739834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.740553 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.740554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.741428 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.741430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.741846 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.741848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.741982 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.742462 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:32.742471 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:32.744280 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:32.745712 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.745719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.745721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.745941 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.746170 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.746171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.746171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.746279 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.746542 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:32.746549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:32.746692 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.746800 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.746961 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260928 12:28:32.747296 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 21840
I20260928 12:28:32.747745 21972 heartbeater.cc:347] Connected to a master server at 127.7.19.126:33465
I20260928 12:28:32.747841 21972 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:32.748005 21972 heartbeater.cc:511] Master 127.7.19.126:33465 requested a full tablet report, sending...
I20260928 12:28:32.748603 21511 ts_manager.cc:194] Registered new tserver with Master: a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
I20260928 12:28:32.749244 21511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:45855
Sep 28 12:28:32 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598512, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.67@KRBTEST.COM
I20260928 12:28:32.754556 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.747733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.7.19.126:33465)
0928 12:28:32.747775 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.747777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.747904 (+ 127us) server_negotiation.cc:407] Connection header received
0928 12:28:32.747978 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.747979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.747989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.748034 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.748638 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.749281 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.750228 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.750469 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.750542 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.751677 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.751678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.751679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.751685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.753360 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.753764 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.753766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.753766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.753834 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.754041 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.754042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.754042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.754165 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.754326 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.754387 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.754462 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:28:32.755491 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:28:32.757103 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.748949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:38349 (local address 127.0.0.1:39106)
0928 12:28:32.748991 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.749000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.749033 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.749329 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.749330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.749474 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:32.749508 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.749509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.750432 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.750433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.751137 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.751140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.751642 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.751643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.751701 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.752103 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:32.752108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:32.753677 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:32.755925 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.755926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.755926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.756162 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.756377 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:32.756378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:32.756378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:32.756443 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:32.756824 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:32.756824 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:32.756864 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.756944 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.757041 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:28:32.757196 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.749034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.7.19.67:38349)
0928 12:28:32.749187 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.749191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.749199 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:28:32.749230 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.749232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.749244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.749425 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:32.749559 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.750354 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.751230 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.751551 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.751640 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.753922 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:32.753928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:32.753929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:32.753935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:32.755820 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.756238 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.756241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.756242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.756303 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:32.756544 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:32.756546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:32.756546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:32.756748 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:32.756838 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.756998 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.757095 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260928 12:28:32.757539 21905 tablet_service.cc:1514] Processing DeleteTablet for tablet 64681a557d124c5280b22698cfc9669e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:32 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:39106
I20260928 12:28:32.758142 21978 tablet_replica.cc:330] stopping tablet replica
I20260928 12:28:32.758272 21978 raft_consensus.cc:2229] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:32.758352 21978 raft_consensus.cc:2258] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:32.758683 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.755833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.7.19.126:33465)
0928 12:28:32.755864 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.755866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.755980 (+ 114us) server_negotiation.cc:407] Connection header received
0928 12:28:32.756037 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.756038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.756072 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.756150 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:28:32.756250 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.756911 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.757843 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.758119 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.758186 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.758352 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:28:32.758463 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:28:32.758567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.758613 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.758635 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:28:32.758773 21978 ts_tablet_manager.cc:1905] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:32.758972 21511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 12:28:32.761261 21978 ts_tablet_manager.cc:1918] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 12:28:32.761279 21511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:28:32.761338 21978 log.cc:1196] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/64681a557d124c5280b22698cfc9669e
I20260928 12:28:32.761592 21978 ts_tablet_manager.cc:1939] T 64681a557d124c5280b22698cfc9669e P a9decdf09ecd489eb277c890b7a1068a: Deleting consensus metadata
I20260928 12:28:32.762058 21495 catalog_manager.cc:4841] TS a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): tablet 64681a557d124c5280b22698cfc9669e (table test-table [id=412fbfa63f0445ee969a9f3782bc853e]) successfully deleted
I20260928 12:28:32.763298 21905 tablet_service.cc:1467] Processing CreateTablet for tablet 93208202362e4cafa538377d024760ce (DEFAULT_TABLE table=test-table [id=dc53a4129750412db8a89d3f9c861433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:32.763300 21633 tablet_service.cc:1467] Processing CreateTablet for tablet 93208202362e4cafa538377d024760ce (DEFAULT_TABLE table=test-table [id=dc53a4129750412db8a89d3f9c861433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:32.763447 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93208202362e4cafa538377d024760ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:32.763448 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93208202362e4cafa538377d024760ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:32.764868 21862 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Bootstrap starting.
I20260928 12:28:32.765103 21591 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap starting.
I20260928 12:28:32.765287 21862 tablet_bootstrap.cc:654] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:32.765534 21591 tablet_bootstrap.cc:654] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:32.765946 21862 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: No bootstrap required, opened a new log
I20260928 12:28:32.765995 21862 ts_tablet_manager.cc:1397] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:32.766089 21591 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: No bootstrap required, opened a new log
I20260928 12:28:32.766161 21591 ts_tablet_manager.cc:1397] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:32.766342 21862 raft_consensus.cc:348] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:32.766422 21862 raft_consensus.cc:374] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:32.766472 21862 raft_consensus.cc:729] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9decdf09ecd489eb277c890b7a1068a, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.766492 21591 raft_consensus.cc:348] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:32.766530 21862 consensus_queue.cc:260] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:32.766577 21591 raft_consensus.cc:374] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:32.766624 21862 ts_tablet_manager.cc:1428] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:32.766637 21591 raft_consensus.cc:729] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ddcabc1a6a4c1f91973d45b7f6af26, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.766695 21591 consensus_queue.cc:260] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:32.766737 21972 heartbeater.cc:503] Master 127.7.19.126:33465 was elected leader, sending a full tablet report...
I20260928 12:28:32.766793 21591 ts_tablet_manager.cc:1428] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:32.767516 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.763033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39887 (local address 127.0.0.1:56048)
0928 12:28:32.763062 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:32.763071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:32.763107 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:32.763494 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:32.763495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:32.763496 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:32.763530 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.763531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.764500 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.764501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.766415 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:32.766419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.767326 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.767327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:32.767389 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.767403 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:28:32.767447 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:28:32.767474 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 12:28:32.767544 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:32.763148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.7.19.66:39887)
0928 12:28:32.763321 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:32.763325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:32.763335 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:28:32.763374 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:32.763381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:32.763393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:32.763491 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:32.763588 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.764383 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.766521 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:32.767255 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:32.767318 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:32.767372 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:32.767458 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:28:32.767486 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
I20260928 12:28:32.768155 21771 tablet_service.cc:1467] Processing CreateTablet for tablet 93208202362e4cafa538377d024760ce (DEFAULT_TABLE table=test-table [id=dc53a4129750412db8a89d3f9c861433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:32.768471 21771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93208202362e4cafa538377d024760ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:32.771219 21982 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Bootstrap starting.
I20260928 12:28:32.771795 21982 tablet_bootstrap.cc:654] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:32.772282 21982 log.cc:824] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:32.773042 21982 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: No bootstrap required, opened a new log
I20260928 12:28:32.773123 21982 ts_tablet_manager.cc:1397] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:32.774415 21982 raft_consensus.cc:348] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:32.774576 21982 raft_consensus.cc:374] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:32.774629 21982 raft_consensus.cc:729] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3d8694d314401eb23e126d6eb9b17f, State: Initialized, Role: FOLLOWER
I20260928 12:28:32.774744 21982 consensus_queue.cc:260] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:32.775032 21982 ts_tablet_manager.cc:1428] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:28:32.775127 21837 heartbeater.cc:503] Master 127.7.19.126:33465 was elected leader, sending a full tablet report...
W20260928 12:28:32.842043 21838 tablet.cc:2369] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:28:32.967495 21700 tablet.cc:2369] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:28:32.988802 21973 tablet.cc:2369] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:28:33.005559 21984 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:33.005734 21984 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:33.006134 21984 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349), f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507)
I20260928 12:28:33.010265 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.006405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:55165 (local address 127.7.19.65:41507)
0928 12:28:33.006536 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:33.006538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:33.006594 (+ 56us) server_negotiation.cc:407] Connection header received
0928 12:28:33.006663 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:33.006664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:33.006694 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:33.006753 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:33.006901 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.007611 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.009193 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.009894 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.009978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:33.010057 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:33.010149 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 12:28:33.010190 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260928 12:28:33.010275 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.006410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:35371 (local address 127.7.19.67:38349)
0928 12:28:33.006457 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:33.006459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:33.006489 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:28:33.006519 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:33.006521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:33.006548 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:33.006625 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:33.006811 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.007623 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.009193 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.009893 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.009978 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:33.010057 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:33.010149 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 12:28:33.010189 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:28:33.010265 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.006321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:38349 (local address 127.7.19.66:35371)
0928 12:28:33.006378 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.006391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:33.006437 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:33.006643 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:33.006644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:33.006646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:33.006714 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:33.006715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.007706 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.007707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:33.009089 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:33.009093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.009984 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.009985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:33.010064 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:33.010066 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:28:33.010124 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:28:33.010168 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":101}
I20260928 12:28:33.010272 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.006341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:41507 (local address 127.7.19.66:55165)
0928 12:28:33.006533 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.006540 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:33.006578 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:33.006773 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:33.006774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:33.006776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:33.006837 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:33.006838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.007706 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.007707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:33.009101 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:33.009102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.009984 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.009985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:33.010061 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:33.010065 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:28:33.010127 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:28:33.010168 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":77,"thread_start_us":100,"threads_started":1}
I20260928 12:28:33.010651 21651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93208202362e4cafa538377d024760ce" candidate_uuid: "1d3d8694d314401eb23e126d6eb9b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" is_pre_election: true
I20260928 12:28:33.010666 21925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93208202362e4cafa538377d024760ce" candidate_uuid: "1d3d8694d314401eb23e126d6eb9b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9decdf09ecd489eb277c890b7a1068a" is_pre_election: true
I20260928 12:28:33.010804 21651 raft_consensus.cc:2454] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3d8694d314401eb23e126d6eb9b17f in term 0.
I20260928 12:28:33.010823 21925 raft_consensus.cc:2454] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3d8694d314401eb23e126d6eb9b17f in term 0.
I20260928 12:28:33.011054 21727 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3d8694d314401eb23e126d6eb9b17f, f8ddcabc1a6a4c1f91973d45b7f6af26; no voters:
I20260928 12:28:33.011263 21984 raft_consensus.cc:2781] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:28:33.011343 21984 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:28:33.011369 21984 raft_consensus.cc:3037] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:33.012087 21984 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:33.012234 21984 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [CANDIDATE]: Term 1 election: Requested vote from peers a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349), f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507)
I20260928 12:28:33.012370 21651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93208202362e4cafa538377d024760ce" candidate_uuid: "1d3d8694d314401eb23e126d6eb9b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
I20260928 12:28:33.012398 21925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93208202362e4cafa538377d024760ce" candidate_uuid: "1d3d8694d314401eb23e126d6eb9b17f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9decdf09ecd489eb277c890b7a1068a"
I20260928 12:28:33.012450 21651 raft_consensus.cc:3037] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:33.012461 21925 raft_consensus.cc:3037] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:33.013262 21651 raft_consensus.cc:2454] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3d8694d314401eb23e126d6eb9b17f in term 1.
I20260928 12:28:33.013262 21925 raft_consensus.cc:2454] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3d8694d314401eb23e126d6eb9b17f in term 1.
I20260928 12:28:33.013440 21724 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3d8694d314401eb23e126d6eb9b17f, a9decdf09ecd489eb277c890b7a1068a; no voters:
I20260928 12:28:33.013556 21984 raft_consensus.cc:2781] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:33.013720 21984 raft_consensus.cc:686] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 1 LEADER]: Becoming Leader. State: Replica: 1d3d8694d314401eb23e126d6eb9b17f, State: Running, Role: LEADER
I20260928 12:28:33.013830 21984 consensus_queue.cc:237] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:33.014672 21510 catalog_manager.cc:5495] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66). New cstate: current_term: 1 leader_uuid: "1d3d8694d314401eb23e126d6eb9b17f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } health_report { overall_health: UNKNOWN } } }
I20260928 12:28:33.049850 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.047688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.7.19.66:39887)
0928 12:28:33.047754 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:33.047755 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:33.047781 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:28:33.047816 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:33.047817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:33.047837 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:33.047890 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:28:33.048014 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.048480 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.049163 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:33.049344 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:33.049394 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:33.049515 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:28:33.049610 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:28:33.049671 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:33.049710 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 12:28:33.049727 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:28:33.051749 21925 raft_consensus.cc:1261] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Refusing update from remote peer 1d3d8694d314401eb23e126d6eb9b17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:33.051749 21651 raft_consensus.cc:1261] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Refusing update from remote peer 1d3d8694d314401eb23e126d6eb9b17f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:33.052137 21987 consensus_queue.cc:1035] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:33.052301 21984 consensus_queue.cc:1035] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:33.054462 21990 mvcc.cc:205] Tried to move back new op lower bound from 7334291509456723968 to 7334291509305131008. Current Snapshot: MvccSnapshot[applied={T|T < 7334291509456723968}]
I20260928 12:28:33.054802 21992 mvcc.cc:205] Tried to move back new op lower bound from 7334291509456723968 to 7334291509305131008. Current Snapshot: MvccSnapshot[applied={T|T < 7334291509456723968}]
I20260928 12:28:33.056773 21510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:33.056879 21510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:33.057646 21510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Sending DeleteTablet for 3 replicas of tablet 93208202362e4cafa538377d024760ce
I20260928 12:28:33.057958 21905 tablet_service.cc:1514] Processing DeleteTablet for tablet 93208202362e4cafa538377d024760ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:33 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:39106
I20260928 12:28:33.057958 21771 tablet_service.cc:1514] Processing DeleteTablet for tablet 93208202362e4cafa538377d024760ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:33 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:56048
I20260928 12:28:33.058058 21978 tablet_replica.cc:330] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: stopping tablet replica
I20260928 12:28:33.058125 21978 raft_consensus.cc:2229] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:33.058121 21633 tablet_service.cc:1514] Processing DeleteTablet for tablet 93208202362e4cafa538377d024760ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:33 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:55348
I20260928 12:28:33.058164 21978 raft_consensus.cc:2258] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:33.058161 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21549
I20260928 12:28:33.058818 22000 tablet_replica.cc:330] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: stopping tablet replica
I20260928 12:28:33.058825 21978 ts_tablet_manager.cc:1905] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:33.058974 22000 raft_consensus.cc:2229] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:28:33.059103 22000 raft_consensus.cc:2258] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:33.059571 22000 ts_tablet_manager.cc:1905] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:33.060364 21978 ts_tablet_manager.cc:1918] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:33.060427 21978 log.cc:1196] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/93208202362e4cafa538377d024760ce
I20260928 12:28:33.060655 21978 ts_tablet_manager.cc:1939] T 93208202362e4cafa538377d024760ce P a9decdf09ecd489eb277c890b7a1068a: Deleting consensus metadata
I20260928 12:28:33.061630 21495 catalog_manager.cc:4841] TS a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): tablet 93208202362e4cafa538377d024760ce (table test-table [id=dc53a4129750412db8a89d3f9c861433]) successfully deleted
I20260928 12:28:33.061825 22000 ts_tablet_manager.cc:1918] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:33.061935 22000 log.cc:1196] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/93208202362e4cafa538377d024760ce
I20260928 12:28:33.062388 22000 ts_tablet_manager.cc:1939] T 93208202362e4cafa538377d024760ce P 1d3d8694d314401eb23e126d6eb9b17f: Deleting consensus metadata
I20260928 12:28:33.062819 21498 catalog_manager.cc:4841] TS 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): tablet 93208202362e4cafa538377d024760ce (table test-table [id=dc53a4129750412db8a89d3f9c861433]) successfully deleted
W20260928 12:28:33.065038 21498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:41507 (error 108)
I20260928 12:28:33.065449 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21704
I20260928 12:28:33.065500 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.065376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:41507 (local address 127.0.0.1:55350)
0928 12:28:33.065423 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.065465 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:41507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 12:28:33.066321 21498 catalog_manager.cc:4584] TS f8ddcabc1a6a4c1f91973d45b7f6af26 (127.7.19.65:41507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93208202362e4cafa538377d024760ce: Network error: Client connection negotiation failed: client connection to 127.7.19.65:41507: connect: Connection refused (error 111)
I20260928 12:28:33.070048 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21840
I20260928 12:28:33.074437 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 21478
I20260928 12:28:33.079764 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:33465
--webserver_interface=127.7.19.126
--webserver_port=36657
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:33465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:33.165158 22002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:33.165369 22002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:33.165392 22002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:33.166992 22002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:33.167053 22002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:33.167079 22002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:33.167102 22002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:33.167131 22002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:33.169099 22002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:33465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=36657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:33.169364 22002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:33.169616 22002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:33.174476 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.174475 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.174506 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:33.174700 22002 server_base.cc:1034] running on GCE node
I20260928 12:28:33.175313 22002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:33.175654 22002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:33.176766 22002 hybrid_clock.cc:648] HybridClock initialized: now 1790598513176751 us; error 31 us; skew 500 ppm
Sep 28 12:28:33 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:33.179397 22002 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:33.179930 22002 webserver.cc:460] Webserver started at http://127.7.19.126:36657/ using document root <none> and password file <none>
I20260928 12:28:33.180181 22002 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:33.180261 22002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:33.181794 22002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:33.182466 22017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.182655 22002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:28:33.182765 22002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "f3eeef8a2d454e0ebc90c732be175bd5"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "87b7a645f9c61d94a69f00854a317809"
server_key_iv: "9614616ec0a2cb2e984439d21b3a8691"
server_key_version: "encryptionkey@0"
I20260928 12:28:33.183144 22002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:33.203058 22002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:33.205297 22002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:33.205483 22002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:33.209841 22002 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:33465
I20260928 12:28:33.209874 22069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:33465 every 8 connection(s)
I20260928 12:28:33.210275 22002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:33.211804 22070 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:28:33.212474 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap starting.
I20260928 12:28:33.214888 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22002
I20260928 12:28:33.215186 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:41507
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:33465
--webserver_port=39461
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:33.221298 22070 log.cc:824] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:33.223489 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:28:33.223814 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5: Bootstrap complete.
I20260928 12:28:33.226084 22070 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:33.226316 22070 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3eeef8a2d454e0ebc90c732be175bd5, State: Initialized, Role: FOLLOWER
I20260928 12:28:33.226490 22070 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:33.226573 22070 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:33.226627 22070 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:33.226682 22070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:28:33.227672 22070 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:33.227793 22070 leader_election.cc:302] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3eeef8a2d454e0ebc90c732be175bd5; no voters:
I20260928 12:28:33.227996 22070 leader_election.cc:288] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 12:28:33.228034 22074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:28:33.228147 22074 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [term 3 LEADER]: Becoming Leader. State: Replica: f3eeef8a2d454e0ebc90c732be175bd5, State: Running, Role: LEADER
I20260928 12:28:33.228242 22074 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } }
I20260928 12:28:33.228284 22070 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:33.228515 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } } }
I20260928 12:28:33.228753 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:33.228693 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3eeef8a2d454e0ebc90c732be175bd5. Latest consensus state: current_term: 3 leader_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3eeef8a2d454e0ebc90c732be175bd5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 33465 } } }
I20260928 12:28:33.229017 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3eeef8a2d454e0ebc90c732be175bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:33.229171 22083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:33.229908 22083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:33.230031 22083 catalog_manager.cc:1259] Loaded cluster ID: 59993feec19642a789df02a5176ed15b
I20260928 12:28:33.230070 22083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 12:28:33.230329 22083 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f223b03e9ad google::LogMessage::Fail() at ??:0
@ 0x7f223b04188a google::LogMessage::SendToLog() at ??:0
@ 0x7f223b03e4de google::LogMessage::Flush() at ??:0
@ 0x7f223b03fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2242f5f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2242f688b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f223cf0c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f223cf01954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f223dd656db start_thread at ??:0
@ 0x7f2239b3c61f clone at ??:0
W20260928 12:28:33.308251 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:33.308482 22072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:33.308521 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:33.309784 22072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:33.309844 22072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:33.309891 22072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:28:33.311396 22072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:41507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=39461
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:33.311676 22072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:33.311928 22072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:33.317034 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.317058 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.317114 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:33.317350 22072 server_base.cc:1034] running on GCE node
I20260928 12:28:33.317574 22072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:33.317844 22072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:33.319029 22072 hybrid_clock.cc:648] HybridClock initialized: now 1790598513319006 us; error 39 us; skew 500 ppm
Sep 28 12:28:33 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:33.322322 22072 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:28:33.322818 22072 webserver.cc:460] Webserver started at http://127.7.19.65:39461/ using document root <none> and password file <none>
I20260928 12:28:33.323066 22072 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:33.323139 22072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:33.324594 22072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:28:33.325388 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.325627 22072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:33.325732 22072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "f63a47250443aa51bb5ce9955484773b"
server_key_iv: "776afa63387ff87010050384a954e3bc"
server_key_version: "encryptionkey@0"
I20260928 12:28:33.326102 22072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:33.345187 22072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:33.347407 22072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:33.347587 22072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:33.347836 22072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:33.348351 22123 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:28:33.350090 22072 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:28:33.350162 22072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:28:33.350209 22072 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:28:33.350970 22072 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:28:33.351025 22072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:28:33.351099 22123 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap starting.
I20260928 12:28:33.358112 22072 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:41507
I20260928 12:28:33.358222 22230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:41507 every 8 connection(s)
I20260928 12:28:33.358523 22072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:28:33.360744 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22072
I20260928 12:28:33.361024 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:39887
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:33465
--webserver_port=38477
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:28:33.362995 22123 log.cc:824] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:33.366981 22123 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:28:33.367311 22123 tablet_bootstrap.cc:492] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Bootstrap complete.
I20260928 12:28:33.368583 22123 ts_tablet_manager.cc:1397] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260928 12:28:33.370009 22123 raft_consensus.cc:348] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:33.370338 22123 raft_consensus.cc:729] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ddcabc1a6a4c1f91973d45b7f6af26, State: Initialized, Role: FOLLOWER
I20260928 12:28:33.370515 22123 consensus_queue.cc:260] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:33.370805 22123 ts_tablet_manager.cc:1428] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260928 12:28:33.449970 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:33.450286 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:33.450284 22233 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:28:33.359216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:47071)
0928 12:28:33.359393 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.359480 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:33.359900 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:33.450061 (+ 90161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260928 12:28:33.450330 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:33.450590 22231 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:33465 (0 consecutive failures): Network error: Failed to ping master at 127.7.19.126:33465: Client connection negotiation failed: client connection to 127.7.19.126:33465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:28:33.450986 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.450808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:51103)
0928 12:28:33.450889 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.450945 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 12:28:33.451474 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.451367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:48151)
0928 12:28:33.451419 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.451445 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 12:28:33.451624 22231 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:28:33.452138 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:33.452203 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:33.452257 22234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:28:33.453823 22234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:39887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=38477
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:33.454095 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:33.454329 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:33.459599 22242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.459616 22241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.459625 22244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:33.459653 22234 server_base.cc:1034] running on GCE node
I20260928 12:28:33.460096 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:33.460343 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:33.461515 22234 hybrid_clock.cc:648] HybridClock initialized: now 1790598513461510 us; error 35 us; skew 500 ppm
Sep 28 12:28:33 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:33.464104 22234 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:28:33.464568 22234 webserver.cc:460] Webserver started at http://127.7.19.66:38477/ using document root <none> and password file <none>
I20260928 12:28:33.464829 22234 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:33.464902 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:33.466220 22234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:33.466786 22251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.466990 22234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:28:33.467097 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "1d3d8694d314401eb23e126d6eb9b17f"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "b4980f313032c067f8edaf2e9dc54975"
server_key_iv: "fcadc03fcb7955a4bed5d9ce4eb6d77e"
server_key_version: "encryptionkey@0"
I20260928 12:28:33.467476 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:33.485769 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:33.487990 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:33.488163 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:33.488415 22234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:33.488808 22234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:33.488865 22234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.488900 22234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:33.488927 22234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.494850 22234 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:39887
I20260928 12:28:33.494903 22364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:39887 every 8 connection(s)
I20260928 12:28:33.495232 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:28:33.496191 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.495833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:45787)
0928 12:28:33.495981 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.496052 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
W20260928 12:28:33.496598 22365 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:33465 (0 consecutive failures): Network error: Failed to ping master at 127.7.19.126:33465: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
I20260928 12:28:33.496914 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.496832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:50241)
0928 12:28:33.496872 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.496898 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 12:28:33.497157 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.497108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:53521)
0928 12:28:33.497129 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.497146 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 12:28:33.497278 22365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:28:33.499341 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22234
I20260928 12:28:33.499630 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:38349
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:33465
--webserver_port=43837
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:33.583638 22368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:33.583853 22368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:33.583889 22368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:33.585058 22368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:33.585121 22368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:33.585165 22368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:28:33.586563 22368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:38349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=43837
--tserver_master_addrs=127.7.19.126:33465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:33.586820 22368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:33.587033 22368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:33.591861 22373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.591871 22374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:33.591871 22376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:33.592156 22368 server_base.cc:1034] running on GCE node
I20260928 12:28:33.592386 22368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:33.592592 22368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:33.593787 22368 hybrid_clock.cc:648] HybridClock initialized: now 1790598513593784 us; error 59 us; skew 500 ppm
Sep 28 12:28:33 dist-test-slave-cb7x krb5kdc[20935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:33.596241 22368 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:28:33.596761 22368 webserver.cc:460] Webserver started at http://127.7.19.67:43837/ using document root <none> and password file <none>
I20260928 12:28:33.597007 22368 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:33.597083 22368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:33.598388 22368 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:28:33.598968 22383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.599187 22368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:28:33.599260 22368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "a9decdf09ecd489eb277c890b7a1068a"
format_stamp: "Formatted at 2026-09-28 12:28:31 on dist-test-slave-cb7x"
server_key: "3a815354654c78642f39e734aeb37366"
server_key_iv: "f75219f6b9cb6391777327372d8b2819"
server_key_version: "encryptionkey@0"
I20260928 12:28:33.599607 22368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:33.628919 22368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:33.630815 22368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:33.630980 22368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:33.631218 22368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:33.631532 22368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:33.631585 22368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.631626 22368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:33.631657 22368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:33.637225 22368 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:38349
I20260928 12:28:33.637271 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:38349 every 8 connection(s)
I20260928 12:28:33.637606 22368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:28:33.638619 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.638234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:50071)
0928 12:28:33.638411 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.638483 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
W20260928 12:28:33.639050 22497 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:33465 (0 consecutive failures): Network error: Failed to ping master at 127.7.19.126:33465: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
I20260928 12:28:33.639310 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.639222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:58283)
0928 12:28:33.639266 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.639292 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 12:28:33.639569 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:33.639494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:58121)
0928 12:28:33.639534 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:33.639556 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:28:33.639683 22497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:28:33.644724 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22368
I20260928 12:28:34.452191 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.451969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:50027)
0928 12:28:34.452094 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:34.452132 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 12:28:34.497789 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.497592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:57733)
0928 12:28:34.497707 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:34.497735 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260928 12:28:34.614108 22503 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:34.614269 22503 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:34.614632 22503 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887), a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
Sep 28 12:28:34 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.67@KRBTEST.COM
Sep 28 12:28:34 dist-test-slave-cb7x krb5kdc[20935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598513, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.66@KRBTEST.COM
I20260928 12:28:34.632598 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.614870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:43215 (local address 127.7.19.67:38349)
0928 12:28:34.615026 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:34.615039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:34.615064 (+ 25us) server_negotiation.cc:407] Connection header received
0928 12:28:34.615684 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:34.615687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:34.615767 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:34.615920 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:34.617047 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.618482 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.620167 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.620846 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.620960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:34.628709 (+ 7749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:34.628714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:34.628715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:34.628728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:34.630667 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631222 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:34.631225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:34.631226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:34.631336 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631625 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:34.631626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:34.631626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:34.631916 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:34.631976 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:34.632376 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 12:28:34.632502 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260928 12:28:34.632643 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.614828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:39887 (local address 127.7.19.65:49855)
0928 12:28:34.614966 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:34.614977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:34.615036 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:34.616007 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:34.616011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:34.616582 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:34.616911 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:34.616920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.618611 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.618612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:34.620068 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:34.620071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.620949 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.620951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:34.621100 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:34.626541 (+ 5441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:34.626554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:34.628924 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:34.631099 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:34.631100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:34.631328 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:34.631631 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:34.631632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:34.631692 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:34.632121 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:34.632122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:34.632183 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 12:28:34.632269 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:28:34.632483 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":92,"thread_start_us":69,"threads_started":1}
I20260928 12:28:34.632733 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.614961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:49855 (local address 127.7.19.66:39887)
0928 12:28:34.615098 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:34.615106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:34.615133 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:28:34.615722 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:34.615725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:34.615767 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:34.615920 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:34.617047 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.618481 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.620167 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.620846 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.620960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:34.629071 (+ 8111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:34.629082 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:34.629083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:34.629096 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:34.631007 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631421 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:34.631425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:34.631425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:34.631527 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631762 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:34.631763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:34.631763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:34.632044 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:34.632133 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:34.632354 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 12:28:34.632517 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260928 12:28:34.633191 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.614787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:38349 (local address 127.7.19.65:43215)
0928 12:28:34.614877 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:34.614890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:34.614950 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:34.616007 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:34.616011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:34.616582 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:34.616912 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:34.616920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.618621 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.618622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:34.620068 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:34.620071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:34.620950 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:34.620951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:34.621100 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:34.626541 (+ 5441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:34.626554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:34.628498 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:34.630758 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:34.630766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:34.630768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:34.631124 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:34.631413 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:34.631414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:34.631415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:34.631556 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:34.631990 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:34.631996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:34.632184 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 12:28:34.632303 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 12:28:34.632572 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":229}
W20260928 12:28:34.633680 22120 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
W20260928 12:28:34.634001 22117 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
I20260928 12:28:34.634043 22117 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8ddcabc1a6a4c1f91973d45b7f6af26; no voters: 1d3d8694d314401eb23e126d6eb9b17f, a9decdf09ecd489eb277c890b7a1068a
I20260928 12:28:34.634150 22503 raft_consensus.cc:2726] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:28:34.640187 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:34.639983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:39035)
0928 12:28:34.640116 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:34.640158 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260928 12:28:35.452984 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:35.452771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:33023)
0928 12:28:35.452897 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:35.452933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260928 12:28:35.498430 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:35.498239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:39407)
0928 12:28:35.498351 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:35.498389 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 12:28:35.640892 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:35.640655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:40907)
0928 12:28:35.640818 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:35.640854 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260928 12:28:36.453697 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:36.453421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:37827)
0928 12:28:36.453615 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:36.453651 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":134,"threads_started":1}
I20260928 12:28:36.499228 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:36.498972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:43945)
0928 12:28:36.499153 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:36.499188 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":124,"threads_started":1}
I20260928 12:28:36.641571 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:36.641359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:50933)
0928 12:28:36.641497 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:36.641529 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 12:28:36.784291 22514 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:36.784394 22514 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:36.784559 22514 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887), a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
W20260928 12:28:36.784992 22120 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
W20260928 12:28:36.785082 22117 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
I20260928 12:28:36.785202 22117 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8ddcabc1a6a4c1f91973d45b7f6af26; no voters: 1d3d8694d314401eb23e126d6eb9b17f, a9decdf09ecd489eb277c890b7a1068a
I20260928 12:28:36.785275 22514 raft_consensus.cc:2726] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:28:37.454425 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:37.454199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:40663)
0928 12:28:37.454338 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:37.454377 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 12:28:37.500016 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:37.499797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:44109)
0928 12:28:37.499925 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:37.499970 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 12:28:37.642210 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:37.641997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:58807)
0928 12:28:37.642134 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:37.642168 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260928 12:28:38.455144 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:38.454920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:56335)
0928 12:28:38.455062 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:38.455100 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260928 12:28:38.500782 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:38.500490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:52549)
0928 12:28:38.500698 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:38.500741 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260928 12:28:38.642881 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:38.642669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:38685)
0928 12:28:38.642805 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:38.642840 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260928 12:28:39.455847 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:39.455596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:54449)
0928 12:28:39.455761 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:39.455800 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260928 12:28:39.501443 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:39.501201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:57947)
0928 12:28:39.501341 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:39.501389 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 12:28:39.643682 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:39.643461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:37203)
0928 12:28:39.643594 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:39.643636 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260928 12:28:39.977803 22524 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:39.977952 22524 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:39.978155 22524 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887), a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
W20260928 12:28:39.978672 22117 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
W20260928 12:28:39.978760 22120 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
I20260928 12:28:39.978807 22120 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8ddcabc1a6a4c1f91973d45b7f6af26; no voters: 1d3d8694d314401eb23e126d6eb9b17f, a9decdf09ecd489eb277c890b7a1068a
I20260928 12:28:39.978881 22524 raft_consensus.cc:2726] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:28:40.456633 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:40.456416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:42371)
0928 12:28:40.456554 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:40.456585 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 12:28:40.502241 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:40.501963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:35147)
0928 12:28:40.502161 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:40.502201 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":119,"threads_started":1}
I20260928 12:28:40.644367 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:40.644172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:47553)
0928 12:28:40.644289 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:40.644326 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260928 12:28:41.457332 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:41.457096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:46399)
0928 12:28:41.457246 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:41.457285 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260928 12:28:41.502967 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:41.502741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:52823)
0928 12:28:41.502883 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:41.502926 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260928 12:28:41.645089 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:41.644897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:38259)
0928 12:28:41.645009 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:41.645044 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 12:28:42.458052 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:42.457819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:33175)
0928 12:28:42.457969 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:42.458007 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260928 12:28:42.503667 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:42.503470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:38345)
0928 12:28:42.503586 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:42.503625 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 12:28:42.645743 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:42.645541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:59143)
0928 12:28:42.645661 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:42.645698 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 12:28:43.085781 22534 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:43.085877 22534 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:43.086107 22534 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887), a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
W20260928 12:28:43.086544 22117 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
W20260928 12:28:43.086632 22120 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
I20260928 12:28:43.086707 22120 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8ddcabc1a6a4c1f91973d45b7f6af26; no voters: 1d3d8694d314401eb23e126d6eb9b17f, a9decdf09ecd489eb277c890b7a1068a
I20260928 12:28:43.086777 22534 raft_consensus.cc:2726] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:28:43.458812 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:43.458578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:46903)
0928 12:28:43.458723 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:43.458766 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260928 12:28:43.504407 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:43.504168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:46833)
0928 12:28:43.504310 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:43.504346 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260928 12:28:43.646409 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:43.646199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:36629)
0928 12:28:43.646326 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:43.646363 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 12:28:44.459705 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:44.459422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:59709)
0928 12:28:44.459568 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:44.459619 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260928 12:28:44.505270 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:44.505038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:44077)
0928 12:28:44.505186 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:44.505224 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 12:28:44.647080 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:44.646877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:54015)
0928 12:28:44.646995 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:44.647035 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 12:28:45.460511 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:45.460263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:48855)
0928 12:28:45.460419 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:45.460459 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260928 12:28:45.505957 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:45.505727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:47193)
0928 12:28:45.505873 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:45.505911 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260928 12:28:45.647907 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:45.647679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:41843)
0928 12:28:45.647823 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:45.647864 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 12:28:46.461287 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:46.461045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:42247)
0928 12:28:46.461198 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:46.461240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260928 12:28:46.506641 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:46.506435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:35053)
0928 12:28:46.506559 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:46.506591 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260928 12:28:46.648603 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:46.648390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:32957)
0928 12:28:46.648520 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:46.648559 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260928 12:28:46.819018 22547 raft_consensus.cc:482] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:46.819113 22547 raft_consensus.cc:504] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d8694d314401eb23e126d6eb9b17f" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 39887 } } peers { permanent_uuid: "a9decdf09ecd489eb277c890b7a1068a" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 38349 } } peers { permanent_uuid: "f8ddcabc1a6a4c1f91973d45b7f6af26" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 41507 } }
I20260928 12:28:46.819326 22547 leader_election.cc:288] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887), a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349)
W20260928 12:28:46.819689 22120 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3d8694d314401eb23e126d6eb9b17f (127.7.19.66:39887): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
W20260928 12:28:46.819767 22117 leader_election.cc:341] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9decdf09ecd489eb277c890b7a1068a (127.7.19.67:38349): Not found: Tablet not found: 93208202362e4cafa538377d024760ce
I20260928 12:28:46.819840 22117 leader_election.cc:302] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8ddcabc1a6a4c1f91973d45b7f6af26; no voters: 1d3d8694d314401eb23e126d6eb9b17f, a9decdf09ecd489eb277c890b7a1068a
I20260928 12:28:46.819937 22547 raft_consensus.cc:2726] T 93208202362e4cafa538377d024760ce P f8ddcabc1a6a4c1f91973d45b7f6af26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:28:47.462080 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:47.461840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:47879)
0928 12:28:47.461975 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:47.462016 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 12:28:47.507449 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:47.507206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:37461)
0928 12:28:47.507351 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:47.507393 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 12:28:47.649363 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:47.649104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:57519)
0928 12:28:47.649282 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:47.649319 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260928 12:28:48.462946 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:48.462712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.65:35275)
0928 12:28:48.462857 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:48.462900 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260928 12:28:48.508265 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:48.508028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.66:58263)
0928 12:28:48.508175 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:48.508223 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 12:28:48.648739 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22072
I20260928 12:28:48.650321 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:48.650117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:33465 (local address 127.7.19.67:35353)
0928 12:28:48.650231 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:48.650266 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260928 12:28:48.653640 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22234
I20260928 12:28:48.657814 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22368
2026-09-28T12:28:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18525 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:28:48 dist-test-slave-cb7x krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:28:48 dist-test-slave-cb7x krb5kdc[22556](info): set up 2 sockets
Sep 28 12:28:48 dist-test-slave-cb7x krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260928 12:28:49.697023 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:28:49 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:28:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:28:49.819332 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43431
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:49.901949 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:49.902172 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:49.902240 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:49.903553 22569 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:28:49.903597 22569 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:28:49.903612 22569 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:28:49.903625 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:28:49.903640 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:49.903654 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:49.903668 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:28:49.903681 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:28:49.905332 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:43431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:49.905575 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:49.905776 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:49.911458 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:49.911496 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:49.911558 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:49.911731 22569 server_base.cc:1034] running on GCE node
I20260928 12:28:49.911999 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:49.912249 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:49.913422 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790598529913408 us; error 31 us; skew 500 ppm
Sep 28 12:28:49 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598529, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:49.915966 22569 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:28:49.916405 22569 webserver.cc:460] Webserver started at http://127.7.19.126:44221/ using document root <none> and password file <none>
I20260928 12:28:49.916630 22569 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:49.916719 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:49.916870 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:49.917956 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "1c3e565e17ea470da03f3eff901e036e"
format_stamp: "Formatted at 2026-09-28 12:28:49 on dist-test-slave-cb7x"
server_key: "e33165cf4e68d557ee065d060597b037"
server_key_iv: "bcf75c243b14c5bf055c0bdbb82c28ac"
server_key_version: "encryptionkey@0"
I20260928 12:28:49.918329 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "1c3e565e17ea470da03f3eff901e036e"
format_stamp: "Formatted at 2026-09-28 12:28:49 on dist-test-slave-cb7x"
server_key: "e33165cf4e68d557ee065d060597b037"
server_key_iv: "bcf75c243b14c5bf055c0bdbb82c28ac"
server_key_version: "encryptionkey@0"
I20260928 12:28:49.919637 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:28:49.920337 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:49.920507 22569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:28:49.920607 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "1c3e565e17ea470da03f3eff901e036e"
format_stamp: "Formatted at 2026-09-28 12:28:49 on dist-test-slave-cb7x"
server_key: "e33165cf4e68d557ee065d060597b037"
server_key_iv: "bcf75c243b14c5bf055c0bdbb82c28ac"
server_key_version: "encryptionkey@0"
I20260928 12:28:49.920712 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:49.938798 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:49.941197 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:49.941365 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:49.945293 22569 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:43431
I20260928 12:28:49.945338 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:43431 every 8 connection(s)
I20260928 12:28:49.945729 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:28:49.946658 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:49.949153 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap starting.
I20260928 12:28:49.949815 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:49.950176 22637 log.cc:824] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:49.950889 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: No bootstrap required, opened a new log
I20260928 12:28:49.952602 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:28:49.952771 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:49.952798 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3e565e17ea470da03f3eff901e036e, State: Initialized, Role: FOLLOWER
I20260928 12:28:49.952996 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:28:49.953068 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:28:49.953114 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:28:49.953171 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:49.953727 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:28:49.953846 22637 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3e565e17ea470da03f3eff901e036e; no voters:
I20260928 12:28:49.954054 22637 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:28:49.954126 22640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:49.954270 22640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 LEADER]: Becoming Leader. State: Replica: 1c3e565e17ea470da03f3eff901e036e, State: Running, Role: LEADER
I20260928 12:28:49.954293 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22569
I20260928 12:28:49.954370 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:28:49.954387 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:28:49.954353 22640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:28:49.954658 7245 external_mini_cluster.cc:1428] Setting key c91b4fe56442ff7dc42c772c2fbd9a1d
I20260928 12:28:49.954962 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3e565e17ea470da03f3eff901e036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } } }
I20260928 12:28:49.955049 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:49.954996 22642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3e565e17ea470da03f3eff901e036e. Latest consensus state: current_term: 1 leader_uuid: "1c3e565e17ea470da03f3eff901e036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } } }
I20260928 12:28:49.955255 22642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: This master's current role is: LEADER
I20260928 12:28:49.955420 22649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:28:49.956001 22649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:28:49.957921 22649 catalog_manager.cc:1347] Generated new cluster ID: 7dc6660bc3e14c3a813afe99a6314ab8
I20260928 12:28:49.957988 22649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:28:49 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:49.965857 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:49.955349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.7.19.126:43431)
0928 12:28:49.955573 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:49.955585 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:49.955636 (+ 51us) server_negotiation.cc:407] Connection header received
0928 12:28:49.955827 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:49.955835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:49.955914 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:49.956041 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:49.957095 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:49.958355 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:49.959134 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:49.959577 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:49.959705 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:49.962053 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:49.962057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:49.962057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:49.962069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:49.964292 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:49.964698 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:49.964701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:49.964702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:49.964793 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:49.965001 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:49.965002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:49.965003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:49.965255 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:49.965419 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:49.965491 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:28:49.965635 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260928 12:28:49.968077 22649 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:28:49.968649 22649 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:28:49.975668 22649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Generated new TSK 0
I20260928 12:28:49.975865 22649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:28:50.007615 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43431
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:50.105752 22663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:50.105959 22663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:50.105983 22663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:50.107311 22663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:50.107378 22663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:50.107425 22663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:28:50.108963 22663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:50.109194 22663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:50.109434 22663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:50.114593 22669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:50.114629 22668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:50.114789 22663 server_base.cc:1034] running on GCE node
W20260928 12:28:50.114602 22671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:50.115088 22663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:50.115342 22663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:50.116561 22663 hybrid_clock.cc:648] HybridClock initialized: now 1790598530116544 us; error 38 us; skew 500 ppm
Sep 28 12:28:50 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:50.119186 22663 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:28:50.119618 22663 webserver.cc:460] Webserver started at http://127.7.19.65:35977/ using document root <none> and password file <none>
I20260928 12:28:50.119850 22663 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:50.119920 22663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:50.120082 22663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:50.121169 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "e1095fcc18868a0d33fa17942a704075"
server_key_iv: "f5aecd153d3aa7481bd8997c983e8456"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.121532 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "e1095fcc18868a0d33fa17942a704075"
server_key_iv: "f5aecd153d3aa7481bd8997c983e8456"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.122848 22663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:28:50.123787 22678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.123998 22663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:50.124109 22663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "e1095fcc18868a0d33fa17942a704075"
server_key_iv: "f5aecd153d3aa7481bd8997c983e8456"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.124202 22663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:50.147334 22663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:50.149554 22663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:50.149721 22663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:50.149973 22663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:50.150292 22663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:50.150347 22663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.150386 22663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:50.150415 22663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.156631 22663 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:44029
I20260928 12:28:50.156695 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:44029 every 8 connection(s)
I20260928 12:28:50.157076 22663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:28:50.163793 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22663
Sep 28 12:28:50 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:50.163908 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:28:50.164139 7245 external_mini_cluster.cc:1428] Setting key cb2375e632aca02719d03dbe005a6a5f
I20260928 12:28:50.167699 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.157885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:57357 (local address 127.7.19.126:43431)
0928 12:28:50.157940 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.157943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.158172 (+ 229us) server_negotiation.cc:407] Connection header received
0928 12:28:50.158748 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.158749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.158762 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.158824 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:50.159872 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.160447 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.161603 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.161859 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.161943 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.164560 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:50.164561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:50.164561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:50.164570 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:50.166186 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.166669 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.166670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.166670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.166732 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.166987 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.166988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.166989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.167104 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:50.167177 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.167464 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.167607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:28:50.167840 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.157829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:57357)
0928 12:28:50.158010 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.158065 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.158631 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.158913 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.158917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.159547 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:50.159752 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.159757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.160557 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.160558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.161500 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.161502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.161968 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.161969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.162106 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.162615 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:50.162624 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:50.164342 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:50.166330 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:50.166337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:50.166339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:50.166576 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:50.166801 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:50.166802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:50.166802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:50.166912 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:50.167181 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:50.167187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:50.167323 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.167427 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.167600 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260928 12:28:50.168270 22792 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43431
I20260928 12:28:50.168365 22792 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:50.168566 22792 heartbeater.cc:511] Master 127.7.19.126:43431 requested a full tablet report, sending...
I20260928 12:28:50.169101 22601 ts_manager.cc:194] Registered new tserver with Master: 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029)
I20260928 12:28:50.169714 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:57357
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:28:50.189810 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43431
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:50.271301 22797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:50.271513 22797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:50.271538 22797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:50.272619 22797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:50.272652 22797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:50.272742 22797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:28:50.274124 22797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:50.274299 22797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:50.274468 22797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:28:50.279436 22797 server_base.cc:1034] running on GCE node
W20260928 12:28:50.279480 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:50.279455 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:50.279435 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:50.279816 22797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:50.280061 22797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:50.281255 22797 hybrid_clock.cc:648] HybridClock initialized: now 1790598530281256 us; error 41 us; skew 500 ppm
Sep 28 12:28:50 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:50.283766 22797 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:28:50.284200 22797 webserver.cc:460] Webserver started at http://127.7.19.66:41335/ using document root <none> and password file <none>
I20260928 12:28:50.284432 22797 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:50.284483 22797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:50.284638 22797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:50.285694 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "3065978401b44167bbfd7e5e55633290"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "93c16668c2e39b3cf7c2a91747521c3b"
server_key_iv: "166bd6caaed5f23a80c33b4e37b02cdb"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.286051 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "3065978401b44167bbfd7e5e55633290"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "93c16668c2e39b3cf7c2a91747521c3b"
server_key_iv: "166bd6caaed5f23a80c33b4e37b02cdb"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.287402 22797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:28:50.288177 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.288405 22797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:50.288515 22797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "3065978401b44167bbfd7e5e55633290"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "93c16668c2e39b3cf7c2a91747521c3b"
server_key_iv: "166bd6caaed5f23a80c33b4e37b02cdb"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.288607 22797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:50.296316 22797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:50.298326 22797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:50.298488 22797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:50.298705 22797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:50.299008 22797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:50.299062 22797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.299103 22797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:50.299127 22797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.305219 22797 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:43259
I20260928 12:28:50.305290 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:43259 every 8 connection(s)
I20260928 12:28:50.305619 22797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:28:50 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:50.314381 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22797
I20260928 12:28:50.314520 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:28:50.314777 7245 external_mini_cluster.cc:1428] Setting key b9eb4c42e8c9b116dde8833d6d783611
I20260928 12:28:50.315240 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.306316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:52845 (local address 127.7.19.126:43431)
0928 12:28:50.306358 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.306359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.306642 (+ 283us) server_negotiation.cc:407] Connection header received
0928 12:28:50.307167 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.307168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.307180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.307233 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:50.308185 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.308766 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.309786 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.309982 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.310068 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.312169 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:50.312170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:50.312171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:50.312178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:50.313745 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.314101 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.314102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.314103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.314153 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.314439 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.314439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.314440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.314635 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:50.314710 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.315032 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.315128 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:28:50.315407 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.306310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:52845)
0928 12:28:50.306489 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.306555 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.307062 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.307311 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.307317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.307877 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:50.308074 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.308078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.308865 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.308866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.309680 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.309682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.310076 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.310077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.310205 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.310690 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:50.310699 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:50.312071 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:50.313820 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:50.313825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:50.313827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:50.314034 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:50.314228 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:50.314228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:50.314229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:50.314371 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:50.314715 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:50.314722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:50.314876 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.315007 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.315168 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260928 12:28:50.316988 22926 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43431
I20260928 12:28:50.317106 22926 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:50.317317 22926 heartbeater.cc:511] Master 127.7.19.126:43431 requested a full tablet report, sending...
I20260928 12:28:50.317694 22601 ts_manager.cc:194] Registered new tserver with Master: 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259)
I20260928 12:28:50.318226 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:52845
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:28:50.340507 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43431
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:28:50.421657 22931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:28:50.421855 22931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:28:50.421880 22931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:28:50.423002 22931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:28:50.423040 22931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:28:50.423065 22931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:28:50.424530 22931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:28:50.424757 22931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:28:50.424937 22931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:28:50.429926 22939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:50.429934 22937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:28:50.430034 22936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:28:50.430228 22931 server_base.cc:1034] running on GCE node
I20260928 12:28:50.430402 22931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:28:50.430585 22931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:28:50.431736 22931 hybrid_clock.cc:648] HybridClock initialized: now 1790598530431720 us; error 34 us; skew 500 ppm
Sep 28 12:28:50 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:28:50.434478 22931 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:28:50.434975 22931 webserver.cc:460] Webserver started at http://127.7.19.67:35861/ using document root <none> and password file <none>
I20260928 12:28:50.435199 22931 fs_manager.cc:359] Metadata directory not provided
I20260928 12:28:50.435268 22931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:28:50.435420 22931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:28:50.436523 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "3ef24ca1095e2a38ffaf293fb9ae7c3c"
server_key_iv: "ac92ed4d683c0a84782a2721103cad33"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.436919 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "3ef24ca1095e2a38ffaf293fb9ae7c3c"
server_key_iv: "ac92ed4d683c0a84782a2721103cad33"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.438261 22931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:50.439069 22946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.439254 22931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:28:50.439352 22931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "3ef24ca1095e2a38ffaf293fb9ae7c3c"
server_key_iv: "ac92ed4d683c0a84782a2721103cad33"
server_key_version: "encryptionkey@0"
I20260928 12:28:50.439473 22931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:28:50.482030 22931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:28:50.484153 22931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:28:50.484329 22931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:28:50.484618 22931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:28:50.484967 22931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:28:50.485026 22931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.485071 22931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:28:50.485095 22931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:28:50.491048 22931 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:34985
I20260928 12:28:50.491094 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:34985 every 8 connection(s)
I20260928 12:28:50.491480 22931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:28:50.495631 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 22931
I20260928 12:28:50.495745 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:28:50.495963 7245 external_mini_cluster.cc:1428] Setting key 14d8668b23740012d585031593845616
Sep 28 12:28:50 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598530, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:28:50.501310 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.492204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:44641 (local address 127.7.19.126:43431)
0928 12:28:50.492286 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.492288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.492543 (+ 255us) server_negotiation.cc:407] Connection header received
0928 12:28:50.493082 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.493083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.493094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.493147 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:50.494050 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.494556 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.495594 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.495889 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.495988 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.498625 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:50.498626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:50.498627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:50.498634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:50.499862 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.500294 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.500296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.500296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.500352 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.500639 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.500640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.500640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.500793 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:50.500870 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.501141 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.501219 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:28:50.501518 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.492187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:44641)
0928 12:28:50.492362 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.492433 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.492990 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.493201 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.493204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.493767 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:28:50.493964 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.493969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.494643 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.494643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.495477 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.495480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.496000 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.496001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.496179 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.496920 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:28:50.496929 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:28:50.498527 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:28:50.499990 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:50.499997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:50.499999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:50.500216 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:50.500451 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:28:50.500451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:28:50.500452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:28:50.500568 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:28:50.500862 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:28:50.500868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:28:50.501010 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.501121 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.501271 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260928 12:28:50.502029 23060 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43431
I20260928 12:28:50.502113 23060 heartbeater.cc:464] Registering TS with master...
I20260928 12:28:50.502292 23060 heartbeater.cc:511] Master 127.7.19.126:43431 requested a full tablet report, sending...
I20260928 12:28:50.502671 22601 ts_manager.cc:194] Registered new tserver with Master: 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
I20260928 12:28:50.503052 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:44641
I20260928 12:28:50.507768 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:28:50.515102 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.509139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.7.19.126:43431)
0928 12:28:50.509207 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.509209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.509281 (+ 72us) server_negotiation.cc:407] Connection header received
0928 12:28:50.509326 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.509326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.509335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.509375 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:28:50.509929 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.510388 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.511143 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.511323 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.511379 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.512484 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:28:50.512485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:28:50.512486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:28:50.512492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:28:50.513778 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.514146 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.514147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.514148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.514203 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:28:50.514445 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:28:50.514446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:28:50.514446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:28:50.514628 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:28:50.514714 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.514890 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.514958 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 12:28:50.517036 22601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:28:50.518731 22601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:28:50.525449 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.521477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43259 (local address 127.0.0.1:60332)
0928 12:28:50.521651 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.521661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.521704 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.522161 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.522162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.522164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:50.522207 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.522209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.523138 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.523139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.524340 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.524343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.525189 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.525189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.525249 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.525253 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.525302 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.525344 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":54,"thread_start_us":61,"threads_started":1}
I20260928 12:28:50.525465 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.521276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.7.19.65:44029)
0928 12:28:50.521441 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.521447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.521610 (+ 163us) server_negotiation.cc:407] Connection header received
0928 12:28:50.521777 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.521780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.521798 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.521852 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:50.522172 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.522966 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.524254 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.524980 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.525067 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.525127 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.525306 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.525347 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 12:28:50.525450 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.521143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:44029 (local address 127.0.0.1:48578)
0928 12:28:50.521468 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.521538 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.521656 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.521975 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.521978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.521980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:50.522068 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.522071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.523061 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.523062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.524137 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.524139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.525075 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.525076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.525164 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.525177 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.525282 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.525342 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":26,"thread_start_us":48,"threads_started":1}
I20260928 12:28:50.525812 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.521676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.7.19.66:43259)
0928 12:28:50.521935 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.521939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.521948 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:28:50.521988 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.521990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.522004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.522089 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:50.522288 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.523034 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.524429 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.525124 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.525189 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.525240 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.525689 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.525735 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":72,"threads_started":1}
I20260928 12:28:50.526276 22726 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 (DEFAULT_TABLE table=test-table [id=e7b619385e874ecc9978fc0b33f6fbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:50.526638 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2c7f45cb7c4ced96de03b36833f3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:50.526923 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.521311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34985 (local address 127.0.0.1:40978)
0928 12:28:50.521609 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.521621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.521711 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.521975 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.521978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.521982 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:50.522068 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.522070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.524450 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.524451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.525291 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.525293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.526716 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.526717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.526776 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.526779 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.526829 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.526863 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":35,"thread_start_us":73,"threads_started":1}
I20260928 12:28:50.527012 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.521421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.7.19.67:34985)
0928 12:28:50.521599 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.521603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.521678 (+ 75us) server_negotiation.cc:407] Connection header received
0928 12:28:50.521816 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.521819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.521835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.521902 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:50.523060 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.524352 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.525376 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.526551 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.526599 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.526652 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.526847 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.526921 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260928 12:28:50.527644 22994 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 (DEFAULT_TABLE table=test-table [id=e7b619385e874ecc9978fc0b33f6fbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:50.527961 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2c7f45cb7c4ced96de03b36833f3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:50.529333 22860 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 (DEFAULT_TABLE table=test-table [id=e7b619385e874ecc9978fc0b33f6fbfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:28:50.529659 22860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2c7f45cb7c4ced96de03b36833f3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:28:50.529788 23077 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap starting.
I20260928 12:28:50.530443 23077 tablet_bootstrap.cc:654] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:50.530680 23078 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Bootstrap starting.
I20260928 12:28:50.531030 23077 log.cc:824] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:50.531291 23078 tablet_bootstrap.cc:654] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:50.531814 23078 log.cc:824] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:50.531845 23077 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: No bootstrap required, opened a new log
I20260928 12:28:50.531903 23077 ts_tablet_manager.cc:1397] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:28:50.532473 23078 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: No bootstrap required, opened a new log
I20260928 12:28:50.532555 23078 ts_tablet_manager.cc:1397] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:50.533114 23081 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Bootstrap starting.
I20260928 12:28:50.533776 23081 tablet_bootstrap.cc:654] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Neither blocks nor log segments found. Creating new log.
I20260928 12:28:50.534042 23077 raft_consensus.cc:348] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.534241 23077 raft_consensus.cc:374] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:50.534292 23077 raft_consensus.cc:729] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc43a92cd5409a8bdb3b73b0c47ae0, State: Initialized, Role: FOLLOWER
I20260928 12:28:50.534329 23081 log.cc:824] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Log is configured to *not* fsync() on all Append() calls
I20260928 12:28:50.534410 23077 consensus_queue.cc:260] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.534294 23078 raft_consensus.cc:348] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.534520 23078 raft_consensus.cc:374] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:50.534566 23078 raft_consensus.cc:729] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df5d3bb13fd40afb38b1e06c62aa144, State: Initialized, Role: FOLLOWER
I20260928 12:28:50.534683 23077 ts_tablet_manager.cc:1428] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:28:50.534672 23078 consensus_queue.cc:260] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.534813 22792 heartbeater.cc:503] Master 127.7.19.126:43431 was elected leader, sending a full tablet report...
I20260928 12:28:50.534927 23078 ts_tablet_manager.cc:1428] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:28:50.535058 23060 heartbeater.cc:503] Master 127.7.19.126:43431 was elected leader, sending a full tablet report...
I20260928 12:28:50.535075 23081 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: No bootstrap required, opened a new log
I20260928 12:28:50.535127 23081 ts_tablet_manager.cc:1397] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:28:50.536960 23081 raft_consensus.cc:348] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.537109 23081 raft_consensus.cc:374] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:28:50.537159 23081 raft_consensus.cc:729] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065978401b44167bbfd7e5e55633290, State: Initialized, Role: FOLLOWER
I20260928 12:28:50.537269 23081 consensus_queue.cc:260] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.537540 23081 ts_tablet_manager.cc:1428] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 12:28:50.537587 22926 heartbeater.cc:503] Master 127.7.19.126:43431 was elected leader, sending a full tablet report...
W20260928 12:28:50.556169 22927 tablet.cc:2369] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:28:50.589653 23085 raft_consensus.cc:482] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:28:50.589776 23085 raft_consensus.cc:504] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.590078 23085 leader_election.cc:288] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
I20260928 12:28:50.593195 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.590420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:51453 (local address 127.7.19.65:44029)
0928 12:28:50.590482 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.590485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.590495 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:28:50.590525 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.590526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.590539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.590598 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:50.590764 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.591247 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.592338 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.592840 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.592923 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.592935 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.593087 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.593123 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:28:50.593233 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.590301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:44029 (local address 127.7.19.66:51453)
0928 12:28:50.590364 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.590378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.590430 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.590609 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.590610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.590612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:50.590684 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.590685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.591338 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.591339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.592245 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.592248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.592922 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.592923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.593016 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.593019 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.593078 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.593122 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 12:28:50.593257 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.590490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49755 (local address 127.7.19.67:34985)
0928 12:28:50.590579 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.590581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.590691 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:28:50.590766 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.590767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.590784 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.590846 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:28:50.591010 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.591462 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.592481 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.592882 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.592966 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.592975 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.593141 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.593179 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 12:28:50.593346 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.590417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34985 (local address 127.7.19.66:49755)
0928 12:28:50.590623 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.590650 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.590703 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.590849 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:28:50.590851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:28:50.590852 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:28:50.590919 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.590920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.591568 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.591569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.592385 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:28:50.592388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.592978 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.592979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:28:50.593068 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.593071 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:28:50.593135 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:28:50.593175 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":136,"threads_started":1}
I20260928 12:28:50.593587 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2c7f45cb7c4ced96de03b36833f3a7" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" is_pre_election: true
I20260928 12:28:50.593637 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2c7f45cb7c4ced96de03b36833f3a7" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" is_pre_election: true
I20260928 12:28:50.593751 23014 raft_consensus.cc:2454] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 0.
I20260928 12:28:50.593751 22746 raft_consensus.cc:2454] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 0.
I20260928 12:28:50.593950 22814 leader_election.cc:302] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144; no voters:
I20260928 12:28:50.594071 23085 raft_consensus.cc:2781] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:28:50.594122 23085 raft_consensus.cc:482] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:28:50.594146 23085 raft_consensus.cc:3037] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:50.594815 23085 raft_consensus.cc:504] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.594940 23085 leader_election.cc:288] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 election: Requested vote from peers 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
I20260928 12:28:50.595130 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2c7f45cb7c4ced96de03b36833f3a7" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
I20260928 12:28:50.595204 23014 raft_consensus.cc:3037] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:50.595160 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2c7f45cb7c4ced96de03b36833f3a7" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
I20260928 12:28:50.595257 22746 raft_consensus.cc:3037] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:28:50.595978 23014 raft_consensus.cc:2454] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 1.
I20260928 12:28:50.596123 22746 raft_consensus.cc:2454] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 1.
I20260928 12:28:50.596210 22814 leader_election.cc:302] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144; no voters:
I20260928 12:28:50.596297 23085 raft_consensus.cc:2781] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:28:50.596481 23085 raft_consensus.cc:686] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 1 LEADER]: Becoming Leader. State: Replica: 3065978401b44167bbfd7e5e55633290, State: Running, Role: LEADER
I20260928 12:28:50.596581 23085 consensus_queue.cc:237] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:28:50.597455 22601 catalog_manager.cc:5495] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3065978401b44167bbfd7e5e55633290 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "3065978401b44167bbfd7e5e55633290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } health_report { overall_health: HEALTHY } } }
I20260928 12:28:50.621043 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:28:50.618657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.7.19.66:43259)
0928 12:28:50.618740 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 12:28:50.618742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:28:50.618751 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:28:50.618787 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:28:50.618788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:28:50.618808 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:28:50.618881 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:28:50.619023 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.619506 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.620288 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:28:50.620476 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:28:50.620539 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:28:50.620713 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:28:50.620809 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:28:50.620876 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:28:50.620907 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 12:28:50.620924 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 12:28:50.623202 22746 raft_consensus.cc:1261] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Refusing update from remote peer 3065978401b44167bbfd7e5e55633290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:50.623224 23014 raft_consensus.cc:1261] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Refusing update from remote peer 3065978401b44167bbfd7e5e55633290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:28:50.623610 23087 consensus_queue.cc:1035] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:50.623745 23085 consensus_queue.cc:1035] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:28:50.628396 22601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:50.628494 22601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:28:50.629280 22601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Sending DeleteTablet for 3 replicas of tablet 6d2c7f45cb7c4ced96de03b36833f3a7
I20260928 12:28:50.629637 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22663
I20260928 12:28:50.629607 22726 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:50 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:48578
I20260928 12:28:50.629642 22860 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:50 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:60332
I20260928 12:28:50.629669 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:50 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:40978
I20260928 12:28:50.630371 23099 tablet_replica.cc:330] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: stopping tablet replica
I20260928 12:28:50.630505 23099 raft_consensus.cc:2229] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:28:50.630651 23099 raft_consensus.cc:2258] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:28:50.631059 23099 ts_tablet_manager.cc:1905] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:50.632710 23099 ts_tablet_manager.cc:1918] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:50.632817 23099 log.cc:1196] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/6d2c7f45cb7c4ced96de03b36833f3a7
I20260928 12:28:50.633109 23099 ts_tablet_manager.cc:1939] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 3065978401b44167bbfd7e5e55633290: Deleting consensus metadata
I20260928 12:28:50.633489 22586 catalog_manager.cc:4841] TS 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): tablet 6d2c7f45cb7c4ced96de03b36833f3a7 (table test-table [id=e7b619385e874ecc9978fc0b33f6fbfe]) successfully deleted
W20260928 12:28:50.635317 22586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:44029 (error 108)
I20260928 12:28:50.635632 23100 tablet_replica.cc:330] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: stopping tablet replica
I20260928 12:28:50.635742 23100 raft_consensus.cc:2229] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:28:50.635850 23100 raft_consensus.cc:2258] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 12:28:50.636083 23073 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:28:50.635726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:44029 (local address 127.0.0.1:48592)
0928 12:28:50.635770 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:28:50.635780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:28:50.635814 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:28:50.636037 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:44029: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:28:50.636221 22586 catalog_manager.cc:4584] TS 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d2c7f45cb7c4ced96de03b36833f3a7: Network error: Client connection negotiation failed: client connection to 127.7.19.65:44029: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:28:50.636492 23100 ts_tablet_manager.cc:1905] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:28:50.637040 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22797
I20260928 12:28:50.638139 23100 ts_tablet_manager.cc:1918] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:28:50.638195 23100 log.cc:1196] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/6d2c7f45cb7c4ced96de03b36833f3a7
I20260928 12:28:50.638386 23100 ts_tablet_manager.cc:1939] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4df5d3bb13fd40afb38b1e06c62aa144: Deleting consensus metadata
I20260928 12:28:50.638976 22586 catalog_manager.cc:4841] TS 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): tablet 6d2c7f45cb7c4ced96de03b36833f3a7 (table test-table [id=e7b619385e874ecc9978fc0b33f6fbfe]) successfully deleted
I20260928 12:28:50.642243 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22931
I20260928 12:28:50.646773 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 22569
I20260928 12:29:10.651862 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43431
--webserver_interface=127.7.19.126
--webserver_port=44221
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:10.736243 23102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:10.736439 23102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:10.736470 23102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:10.737635 23102 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:29:10.737676 23102 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:29:10.737690 23102 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:29:10.737703 23102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:10.737717 23102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:10.737730 23102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:10.737742 23102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:10.737753 23102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:10.739161 23102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:43431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=44221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:10.739413 23102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:10.739652 23102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:10.748417 23109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:10.748458 23108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:10.748481 23102 server_base.cc:1034] running on GCE node
W20260928 12:29:10.748417 23111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:10.748917 23102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:10.749197 23102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:10.750375 23102 hybrid_clock.cc:648] HybridClock initialized: now 1790598550750376 us; error 52 us; skew 500 ppm
Sep 28 12:29:10 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:10.752980 23102 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:29:10.753481 23102 webserver.cc:460] Webserver started at http://127.7.19.126:44221/ using document root <none> and password file <none>
I20260928 12:29:10.753756 23102 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:10.753836 23102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:10.755251 23102 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 12:29:10.756034 23118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:10.756201 23102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:10.756296 23102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "1c3e565e17ea470da03f3eff901e036e"
format_stamp: "Formatted at 2026-09-28 12:28:49 on dist-test-slave-cb7x"
server_key: "e33165cf4e68d557ee065d060597b037"
server_key_iv: "bcf75c243b14c5bf055c0bdbb82c28ac"
server_key_version: "encryptionkey@0"
I20260928 12:29:10.756786 23102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:10.779665 23102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:10.781749 23102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:10.781893 23102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:10.785885 23102 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:43431
I20260928 12:29:10.785948 23170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:43431 every 8 connection(s)
I20260928 12:29:10.786317 23102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:10.786880 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23102
I20260928 12:29:10.787196 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:44029
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:43431
--webserver_port=35977
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:29:10.788555 23171 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:29:10.789284 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap starting.
I20260928 12:29:10.797804 23171 log.cc:824] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:10.799688 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:29:10.799954 23171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap complete.
I20260928 12:29:10.802249 23171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:10.802489 23171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3e565e17ea470da03f3eff901e036e, State: Initialized, Role: FOLLOWER
I20260928 12:29:10.802687 23171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:10.802778 23171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:10.802834 23171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:10.802881 23171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 1 FOLLOWER]: Advancing to term 2
I20260928 12:29:10.803850 23171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:10.803968 23171 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3e565e17ea470da03f3eff901e036e; no voters:
I20260928 12:29:10.804155 23171 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 12:29:10.804181 23175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 12:29:10.804289 23175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 LEADER]: Becoming Leader. State: Replica: 1c3e565e17ea470da03f3eff901e036e, State: Running, Role: LEADER
I20260928 12:29:10.804384 23175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:10.804422 23171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:10.804636 23176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c3e565e17ea470da03f3eff901e036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } } }
I20260928 12:29:10.804740 23176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:10.804979 23177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3e565e17ea470da03f3eff901e036e. Latest consensus state: current_term: 2 leader_uuid: "1c3e565e17ea470da03f3eff901e036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } } }
I20260928 12:29:10.805063 23177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:10.805157 23179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:10.805897 23179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:10.806015 23179 catalog_manager.cc:1259] Loaded cluster ID: 7dc6660bc3e14c3a813afe99a6314ab8
I20260928 12:29:10.806262 23179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:10.807356 23179 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:10.807524 23179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Loaded TSK: 0
I20260928 12:29:10.821799 23179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Generated new TSK 1
I20260928 12:29:10.821895 23179 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:29:10.873728 23173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:10.873926 23173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:10.873950 23173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:10.875135 23173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:10.875177 23173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:10.875204 23173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:10.876641 23173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:44029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=35977
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:10.876914 23173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:10.877146 23173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:10.882102 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:10.882131 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:10.882266 23173 server_base.cc:1034] running on GCE node
W20260928 12:29:10.882121 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:10.882642 23173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:10.882876 23173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:10.884040 23173 hybrid_clock.cc:648] HybridClock initialized: now 1790598550884027 us; error 32 us; skew 500 ppm
Sep 28 12:29:10 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:10.886703 23173 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:29:10.887274 23173 webserver.cc:460] Webserver started at http://127.7.19.65:35977/ using document root <none> and password file <none>
I20260928 12:29:10.887524 23173 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:10.887609 23173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:10.889439 23173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:29:10.890237 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:10.890453 23173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:10.890550 23173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "e1095fcc18868a0d33fa17942a704075"
server_key_iv: "f5aecd153d3aa7481bd8997c983e8456"
server_key_version: "encryptionkey@0"
I20260928 12:29:10.890974 23173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:10.913388 23173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:10.915951 23173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:10.916138 23173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:10.916388 23173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:10.916908 23215 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:29:10.918345 23173 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:29:10.918407 23173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:29:10.918445 23173 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:29:10.919310 23173 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:29:10.919359 23173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:29:10.919416 23215 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap starting.
I20260928 12:29:10.926191 23173 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:44029
I20260928 12:29:10.926416 23322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:44029 every 8 connection(s)
I20260928 12:29:10.926609 23173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:10.929965 23215 log.cc:824] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:10.932336 23215 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:29:10.932361 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23173
I20260928 12:29:10.932646 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:43259
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:43431
--webserver_port=41335
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:29:10.932657 23215 tablet_bootstrap.cc:492] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap complete.
I20260928 12:29:10.933830 23215 ts_tablet_manager.cc:1397] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
Sep 28 12:29:10 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598550, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:10.935783 23215 raft_consensus.cc:348] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:29:10.936065 23215 raft_consensus.cc:729] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc43a92cd5409a8bdb3b73b0c47ae0, State: Initialized, Role: FOLLOWER
I20260928 12:29:10.936769 23215 consensus_queue.cc:260] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } }
I20260928 12:29:10.937568 23215 ts_tablet_manager.cc:1428] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 12:29:10.940858 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:10.927286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:47035)
0928 12:29:10.927481 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:10.927529 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:10.928100 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:10.929318 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:10.929323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:10.929671 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:10.929929 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:10.929935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.931612 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.931613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:10.932891 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:10.932894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.933655 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.933656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:10.933796 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:10.934389 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:10.934398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:10.936185 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:10.938860 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:10.938866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:10.938868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:10.939166 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:10.939480 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:10.939481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:10.939482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:10.939613 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:10.940065 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:10.940072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:10.940259 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 12:29:10.940392 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 12:29:10.940584 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260928 12:29:10.940886 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:10.927379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:47035 (local address 127.7.19.126:43431)
0928 12:29:10.928825 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:10.928834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:10.928875 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:29:10.929011 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:10.929021 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:10.929095 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:10.929269 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:10.930150 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.931474 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.933002 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.933554 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.934085 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:10.936293 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:10.936297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:10.936298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:10.936311 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:10.938763 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:10.939257 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:10.939260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:10.939260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:10.939403 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:10.939702 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:10.939703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:10.939704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:10.939985 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:10.940068 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:10.940457 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 12:29:10.940602 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":69,"threads_started":1}
I20260928 12:29:10.941761 23323 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43431
I20260928 12:29:10.941869 23323 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:10.942034 23323 heartbeater.cc:511] Master 127.7.19.126:43431 requested a full tablet report, sending...
I20260928 12:29:10.942770 23135 ts_manager.cc:194] Registered new tserver with Master: 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029)
I20260928 12:29:10.943965 23135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:47035
I20260928 12:29:10.945097 23323 heartbeater.cc:503] Master 127.7.19.126:43431 was elected leader, sending a full tablet report...
I20260928 12:29:10.952834 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:10.943381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:44029 (local address 127.0.0.1:52240)
0928 12:29:10.948777 (+ 5396us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:10.948814 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:10.948880 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:10.949157 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:10.949161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:10.949163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:10.949219 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:10.949221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.950460 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.950461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:10.951666 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:10.951669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.952543 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.952543 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:10.952603 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:10.952612 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 12:29:10.952735 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 12:29:10.952781 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5365,"thread_start_us":80,"threads_started":1}
I20260928 12:29:10.952850 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:10.943625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.7.19.65:44029)
0928 12:29:10.949000 (+ 5375us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:10.949005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:10.949013 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:29:10.949043 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:10.949045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:10.949059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:10.949138 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:10.949295 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.950375 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.951752 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:10.952464 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:10.952536 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:10.952586 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:10.952750 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:29:10.952786 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5345,"thread_start_us":81,"threads_started":1}
I20260928 12:29:10.953199 23257 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2c7f45cb7c4ced96de03b36833f3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:28:50 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:52240
I20260928 12:29:10.953686 23332 tablet_replica.cc:330] stopping tablet replica
I20260928 12:29:10.953814 23332 raft_consensus.cc:2229] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:10.953879 23332 raft_consensus.cc:2258] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:10.954178 23332 ts_tablet_manager.cc:1905] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:10.956496 23332 ts_tablet_manager.cc:1918] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:10.956571 23332 log.cc:1196] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal/wals/6d2c7f45cb7c4ced96de03b36833f3a7
I20260928 12:29:10.956812 23332 ts_tablet_manager.cc:1939] T 6d2c7f45cb7c4ced96de03b36833f3a7 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Deleting consensus metadata
I20260928 12:29:10.957245 23120 catalog_manager.cc:4841] TS 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029): tablet 6d2c7f45cb7c4ced96de03b36833f3a7 (table test-table [id=e7b619385e874ecc9978fc0b33f6fbfe]) successfully deleted
W20260928 12:29:11.017684 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:11.017915 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:11.017951 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:11.019055 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:11.019114 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:11.019160 23328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:11.020578 23328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:43259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=41335
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:11.020867 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:11.021080 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:11.026047 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:11.026136 23328 server_base.cc:1034] running on GCE node
W20260928 12:29:11.026072 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:11.026033 23338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:11.026420 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:11.026651 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:11.027809 23328 hybrid_clock.cc:648] HybridClock initialized: now 1790598551027784 us; error 53 us; skew 500 ppm
Sep 28 12:29:11 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:11.030351 23328 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:29:11.030776 23328 webserver.cc:460] Webserver started at http://127.7.19.66:41335/ using document root <none> and password file <none>
I20260928 12:29:11.030954 23328 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:11.031013 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:11.032287 23328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:11.032943 23347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.033110 23328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:11.033195 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "3065978401b44167bbfd7e5e55633290"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "93c16668c2e39b3cf7c2a91747521c3b"
server_key_iv: "166bd6caaed5f23a80c33b4e37b02cdb"
server_key_version: "encryptionkey@0"
I20260928 12:29:11.033515 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:11.047953 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:11.050271 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:11.050424 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:11.050669 23328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:11.050963 23328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:11.051012 23328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.051046 23328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:11.051085 23328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.057049 23328 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:43259
I20260928 12:29:11.057124 23460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:43259 every 8 connection(s)
I20260928 12:29:11.057473 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:29:11.059372 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23328
I20260928 12:29:11.059675 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:34985
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:43431
--webserver_port=35861
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 12:29:11 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:11.068953 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.058214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:38423 (local address 127.7.19.126:43431)
0928 12:29:11.058282 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.058284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.058470 (+ 186us) server_negotiation.cc:407] Connection header received
0928 12:29:11.058947 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.058948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.058962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.059014 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:11.060103 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.060981 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.062483 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.062900 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.062977 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.065639 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:11.065641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:11.065641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:11.065651 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:11.067427 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.067857 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.067858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.067859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.067924 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.068174 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.068175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.068176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.068331 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:11.068408 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.068710 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.068812 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:29:11.069000 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.058212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:38423)
0928 12:29:11.058357 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.058391 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:11.058865 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:11.059069 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:11.059072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:11.059740 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:11.060011 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.060016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.061088 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.061089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.062394 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.062397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.063015 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.063017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.063137 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.063835 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:11.063850 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:11.065540 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:11.067549 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:11.067555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:11.067556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:11.067778 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:11.068004 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:11.068005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:11.068005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:11.068105 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:11.068402 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:11.068406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:11.068559 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 12:29:11.068645 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:29:11.068838 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260928 12:29:11.069567 23461 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43431
I20260928 12:29:11.069681 23461 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:11.069868 23461 heartbeater.cc:511] Master 127.7.19.126:43431 requested a full tablet report, sending...
I20260928 12:29:11.070268 23135 ts_manager.cc:194] Registered new tserver with Master: 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259)
I20260928 12:29:11.070645 23135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:38423
W20260928 12:29:11.143436 23464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:11.143640 23464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:11.143664 23464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:11.144863 23464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:11.144932 23464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:11.144979 23464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:11.146377 23464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=35861
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:11.146570 23464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:11.146737 23464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:11.151749 23469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:11.151844 23472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:11.151760 23470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:11.152508 23464 server_base.cc:1034] running on GCE node
I20260928 12:29:11.152755 23464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:11.152998 23464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:11.154160 23464 hybrid_clock.cc:648] HybridClock initialized: now 1790598551154150 us; error 39 us; skew 500 ppm
Sep 28 12:29:11 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:11.156595 23464 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:29:11.157147 23464 webserver.cc:460] Webserver started at http://127.7.19.67:35861/ using document root <none> and password file <none>
I20260928 12:29:11.157337 23464 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:11.157387 23464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:11.158628 23464 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:11.159201 23479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.159368 23464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:29:11.159456 23464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "3ef24ca1095e2a38ffaf293fb9ae7c3c"
server_key_iv: "ac92ed4d683c0a84782a2721103cad33"
server_key_version: "encryptionkey@0"
I20260928 12:29:11.159780 23464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:11.188697 23464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:11.190622 23464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:11.190734 23464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:11.190913 23464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:11.191221 23464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:11.191254 23464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.191277 23464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:11.191322 23464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.197458 23464 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:34985
I20260928 12:29:11.197517 23592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:34985 every 8 connection(s)
I20260928 12:29:11.197842 23464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:29:11 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598551, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:11.205348 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23464
I20260928 12:29:11.207352 23326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.198493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:50007 (local address 127.7.19.126:43431)
0928 12:29:11.198583 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.198585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.198749 (+ 164us) server_negotiation.cc:407] Connection header received
0928 12:29:11.199280 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.199281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.199295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.199357 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:11.200283 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.200852 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.201820 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.202030 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.202113 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.204137 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:11.204138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:11.204139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:11.204147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:11.205940 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.206423 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.206424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.206424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.206474 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.206747 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.206748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.206748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.206879 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:11.206946 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.207184 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.207262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:29:11.207454 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.198462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:50007)
0928 12:29:11.198600 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.198657 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:11.199174 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:11.199427 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:11.199430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:11.199983 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:11.200178 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.200184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.200937 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.200939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.201732 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.201734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.202121 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.202122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.202250 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.202684 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:11.202691 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:11.204043 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:11.206026 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:11.206034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:11.206036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:11.206352 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:11.206538 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:11.206538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:11.206539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:11.206683 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:11.206944 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:11.206950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:11.207063 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 12:29:11.207152 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 12:29:11.207266 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260928 12:29:11.207948 23593 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43431
I20260928 12:29:11.208053 23593 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:11.208223 23593 heartbeater.cc:511] Master 127.7.19.126:43431 requested a full tablet report, sending...
I20260928 12:29:11.208614 23135 ts_manager.cc:194] Registered new tserver with Master: 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
I20260928 12:29:11.209009 23135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:50007
I20260928 12:29:11.212350 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.205777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.7.19.126:43431)
0928 12:29:11.205890 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.205893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.205989 (+ 96us) server_negotiation.cc:407] Connection header received
0928 12:29:11.206057 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.206058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.206068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.206121 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:11.206847 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.207499 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.208538 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.208876 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.208955 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.210128 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:11.210130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:11.210130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:11.210136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:11.211266 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.211596 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.211597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.211597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.211656 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.211844 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.211845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.211846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.211953 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:11.212137 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.212174 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.212250 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260928 12:29:11.213030 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:11.215296 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.213337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.7.19.126:43431)
0928 12:29:11.213371 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.213373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.213449 (+ 76us) server_negotiation.cc:407] Connection header received
0928 12:29:11.213506 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.213506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.213534 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.213595 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:11.213670 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.214087 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.214782 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.214955 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.215002 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.215117 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:11.215152 (+ 35us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 12:29:11.215258 (+ 106us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49242: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:29:11.215355 23597 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49242: token expired
W20260928 12:29:11.215457 23065 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.19.126:43431)
I20260928 12:29:11.220482 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.215685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.7.19.126:43431)
0928 12:29:11.215716 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.215718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.215727 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:11.215761 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.215761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.215773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.215809 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:11.216194 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.216593 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.217268 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.217429 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.217486 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.218429 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:11.218430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:11.218430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:11.218436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:11.219721 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.219981 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.219981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.219981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.220020 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:11.220135 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:11.220135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:11.220136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:11.220236 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:11.220304 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.220382 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.220433 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:29:11.221576 23135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:11.223306 23135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:11.225888 23257 tablet_service.cc:1467] Processing CreateTablet for tablet a3abd4cd5b9446f7b0e1b85868304982 (DEFAULT_TABLE table=test-table [id=b99aebd84260447da108ba6773668e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:11.226099 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3abd4cd5b9446f7b0e1b85868304982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:11.227640 23215 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap starting.
I20260928 12:29:11.227927 23215 tablet_bootstrap.cc:654] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:11.228403 23215 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: No bootstrap required, opened a new log
I20260928 12:29:11.228456 23215 ts_tablet_manager.cc:1397] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:11.228569 23215 raft_consensus.cc:348] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.228633 23215 raft_consensus.cc:374] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:11.228708 23215 raft_consensus.cc:729] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc43a92cd5409a8bdb3b73b0c47ae0, State: Initialized, Role: FOLLOWER
I20260928 12:29:11.228766 23215 consensus_queue.cc:260] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.228849 23215 ts_tablet_manager.cc:1428] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:11.229544 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.225277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43259 (local address 127.0.0.1:49412)
0928 12:29:11.225337 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.225350 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:11.225392 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:11.225898 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:11.225900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:11.225902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:11.225951 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.225953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.227102 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.227103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.228173 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.228175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.229335 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.229336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.229406 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.229409 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:11.229458 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:29:11.229499 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 12:29:11.229614 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.225392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.7.19.66:43259)
0928 12:29:11.225689 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.225694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.225707 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:29:11.225756 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.225759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.225777 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.226036 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:11.226140 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.227010 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.228385 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.229252 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.229336 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.229398 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.229483 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.229524 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260928 12:29:11.230046 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.225413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34985 (local address 127.0.0.1:40660)
0928 12:29:11.225890 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.225901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:11.225957 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:11.226289 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:11.226290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:11.226292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:11.226330 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.226331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.227523 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.227524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.228874 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.228877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.229831 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.229832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.229905 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.229910 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:29:11.229963 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:29:11.229999 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":129,"threads_started":1}
I20260928 12:29:11.230149 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.225687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.7.19.67:34985)
0928 12:29:11.225870 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.225874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.225946 (+ 72us) server_negotiation.cc:407] Connection header received
0928 12:29:11.226061 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.226064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.226110 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.226191 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:11.226410 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.227429 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.228971 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.229751 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.229820 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.229862 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.229996 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.230040 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260928 12:29:11.230543 23395 tablet_service.cc:1467] Processing CreateTablet for tablet a3abd4cd5b9446f7b0e1b85868304982 (DEFAULT_TABLE table=test-table [id=b99aebd84260447da108ba6773668e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:11.230806 23395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3abd4cd5b9446f7b0e1b85868304982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:11.230746 23527 tablet_service.cc:1467] Processing CreateTablet for tablet a3abd4cd5b9446f7b0e1b85868304982 (DEFAULT_TABLE table=test-table [id=b99aebd84260447da108ba6773668e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:11.231019 23527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3abd4cd5b9446f7b0e1b85868304982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:11.233484 23603 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Bootstrap starting.
I20260928 12:29:11.234048 23604 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Bootstrap starting.
I20260928 12:29:11.234328 23603 tablet_bootstrap.cc:654] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:11.234659 23604 tablet_bootstrap.cc:654] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:11.234871 23603 log.cc:824] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:11.235266 23604 log.cc:824] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:11.235730 23603 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: No bootstrap required, opened a new log
I20260928 12:29:11.235800 23603 ts_tablet_manager.cc:1397] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:11.236006 23604 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: No bootstrap required, opened a new log
I20260928 12:29:11.236056 23604 ts_tablet_manager.cc:1397] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:11.237434 23603 raft_consensus.cc:348] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.237488 23604 raft_consensus.cc:348] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.237598 23603 raft_consensus.cc:374] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:11.237601 23604 raft_consensus.cc:374] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:11.237646 23604 raft_consensus.cc:729] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df5d3bb13fd40afb38b1e06c62aa144, State: Initialized, Role: FOLLOWER
I20260928 12:29:11.237646 23603 raft_consensus.cc:729] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065978401b44167bbfd7e5e55633290, State: Initialized, Role: FOLLOWER
I20260928 12:29:11.237767 23604 consensus_queue.cc:260] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.237771 23603 consensus_queue.cc:260] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.238058 23604 ts_tablet_manager.cc:1428] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:11.238131 23461 heartbeater.cc:503] Master 127.7.19.126:43431 was elected leader, sending a full tablet report...
I20260928 12:29:11.238183 23593 heartbeater.cc:503] Master 127.7.19.126:43431 was elected leader, sending a full tablet report...
I20260928 12:29:11.238068 23603 ts_tablet_manager.cc:1428] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:29:11.273136 23607 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:11.273272 23607 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.273586 23607 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985), 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029)
I20260928 12:29:11.277040 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.273805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34985 (local address 127.7.19.66:36945)
0928 12:29:11.273866 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.273879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:11.273929 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:11.274136 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:11.274137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:11.274144 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:11.274226 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.274228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.275036 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.275037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.276169 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.276172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.276763 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.276764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.276844 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.276847 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:11.276926 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:29:11.276967 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 12:29:11.277040 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.273888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:36945 (local address 127.7.19.67:34985)
0928 12:29:11.273946 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.273949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.273961 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:29:11.273995 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.273996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.274010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.274087 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:11.274311 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.274929 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.276274 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.276652 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.276766 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.276777 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.276929 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.276968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:29:11.277413 23547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3abd4cd5b9446f7b0e1b85868304982" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" is_pre_election: true
I20260928 12:29:11.277649 23547 raft_consensus.cc:2454] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 0.
I20260928 12:29:11.277746 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.274036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:50517 (local address 127.7.19.65:44029)
0928 12:29:11.274097 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.274099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.274236 (+ 137us) server_negotiation.cc:407] Connection header received
0928 12:29:11.274268 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.274270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.274284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.274349 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:11.274516 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.275069 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.276263 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.277390 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.277495 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.277506 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.277629 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.277678 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:29:11.277750 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.273941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:44029 (local address 127.7.19.66:50517)
0928 12:29:11.274154 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.274170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:11.274221 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:11.274371 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:11.274372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:11.274373 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:11.274441 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.274443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.275183 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.275184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.276162 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:11.276164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.277480 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.277481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:11.277537 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.277540 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:11.277615 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:29:11.277652 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
I20260928 12:29:11.277925 23349 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144; no voters:
I20260928 12:29:11.278081 23607 raft_consensus.cc:2781] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:11.278147 23607 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:11.278182 23607 raft_consensus.cc:3037] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:11.278282 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3abd4cd5b9446f7b0e1b85868304982" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" is_pre_election: true
I20260928 12:29:11.278403 23277 raft_consensus.cc:2454] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 0.
I20260928 12:29:11.279022 23607 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.279158 23607 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 election: Requested vote from peers 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985), 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029)
I20260928 12:29:11.279310 23547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3abd4cd5b9446f7b0e1b85868304982" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
I20260928 12:29:11.279350 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3abd4cd5b9446f7b0e1b85868304982" candidate_uuid: "3065978401b44167bbfd7e5e55633290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
I20260928 12:29:11.279410 23547 raft_consensus.cc:3037] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:11.279418 23277 raft_consensus.cc:3037] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:11.280359 23547 raft_consensus.cc:2454] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 1.
I20260928 12:29:11.280359 23277 raft_consensus.cc:2454] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3065978401b44167bbfd7e5e55633290 in term 1.
I20260928 12:29:11.280531 23349 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144; no voters:
I20260928 12:29:11.280622 23607 raft_consensus.cc:2781] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:11.280788 23607 raft_consensus.cc:686] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 1 LEADER]: Becoming Leader. State: Replica: 3065978401b44167bbfd7e5e55633290, State: Running, Role: LEADER
I20260928 12:29:11.280885 23607 consensus_queue.cc:237] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:11.281699 23134 catalog_manager.cc:5495] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3065978401b44167bbfd7e5e55633290 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "3065978401b44167bbfd7e5e55633290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } health_report { overall_health: UNKNOWN } } }
I20260928 12:29:11.291345 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.289347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.7.19.66:43259)
0928 12:29:11.289386 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:11.289388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:11.289393 (+ 5us) server_negotiation.cc:407] Connection header received
0928 12:29:11.289427 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:11.289427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:11.289442 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:11.289543 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:11.289625 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.290070 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.290758 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:11.290928 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:11.290977 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:11.291092 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:11.291178 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:11.291243 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:11.291264 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 12:29:11.291280 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:29:11.292953 23547 raft_consensus.cc:1261] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Refusing update from remote peer 3065978401b44167bbfd7e5e55633290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:11.292992 23277 raft_consensus.cc:1261] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Refusing update from remote peer 3065978401b44167bbfd7e5e55633290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:11.293268 23610 consensus_queue.cc:1035] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:11.293394 23607 consensus_queue.cc:1035] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:11.295627 23612 mvcc.cc:205] Tried to move back new op lower bound from 7334291666093375488 to 7334291666047119360. Current Snapshot: MvccSnapshot[applied={T|T < 7334291666093375488}]
I20260928 12:29:11.295917 23614 mvcc.cc:205] Tried to move back new op lower bound from 7334291666093375488 to 7334291666047119360. Current Snapshot: MvccSnapshot[applied={T|T < 7334291666093375488}]
I20260928 12:29:11.297896 23134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:11.297991 23134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:11.298739 23134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Sending DeleteTablet for 3 replicas of tablet a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:11.299021 23395 tablet_service.cc:1514] Processing DeleteTablet for tablet a3abd4cd5b9446f7b0e1b85868304982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:11 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:49412
I20260928 12:29:11.299074 23257 tablet_service.cc:1514] Processing DeleteTablet for tablet a3abd4cd5b9446f7b0e1b85868304982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:11 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:52240
I20260928 12:29:11.299154 23332 tablet_replica.cc:330] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: stopping tablet replica
I20260928 12:29:11.299108 23527 tablet_service.cc:1514] Processing DeleteTablet for tablet a3abd4cd5b9446f7b0e1b85868304982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:11 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260928 12:29:11.299208 23332 raft_consensus.cc:2229] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:11.299269 23332 raft_consensus.cc:2258] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:11.299373 23621 tablet_replica.cc:330] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: stopping tablet replica
I20260928 12:29:11.299391 23622 tablet_replica.cc:330] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: stopping tablet replica
I20260928 12:29:11.299515 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23173
I20260928 12:29:11.299667 23332 ts_tablet_manager.cc:1905] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:11.300175 23622 raft_consensus.cc:2229] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:11.300354 23622 raft_consensus.cc:2258] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:11.300895 23622 ts_tablet_manager.cc:1905] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:11.303061 23622 ts_tablet_manager.cc:1918] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:11.303190 23622 log.cc:1196] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:11.303565 23622 ts_tablet_manager.cc:1939] T a3abd4cd5b9446f7b0e1b85868304982 P 4df5d3bb13fd40afb38b1e06c62aa144: Deleting consensus metadata
I20260928 12:29:11.303905 23621 raft_consensus.cc:2229] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:29:11.304066 23621 raft_consensus.cc:2258] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:11.304214 23120 catalog_manager.cc:4841] TS 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): tablet a3abd4cd5b9446f7b0e1b85868304982 (table test-table [id=b99aebd84260447da108ba6773668e33]) successfully deleted
I20260928 12:29:11.304441 23621 ts_tablet_manager.cc:1905] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:11.306430 23621 ts_tablet_manager.cc:1918] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:11.306504 23621 log.cc:1196] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:11.306792 23621 ts_tablet_manager.cc:1939] T a3abd4cd5b9446f7b0e1b85868304982 P 3065978401b44167bbfd7e5e55633290: Deleting consensus metadata
I20260928 12:29:11.308040 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23328
W20260928 12:29:11.309775 23120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:44029 (error 108)
I20260928 12:29:11.310096 23120 catalog_manager.cc:4841] TS 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): tablet a3abd4cd5b9446f7b0e1b85868304982 (table test-table [id=b99aebd84260447da108ba6773668e33]) successfully deleted
I20260928 12:29:11.310384 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:11.310256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:44029 (local address 127.0.0.1:52246)
0928 12:29:11.310313 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:11.310354 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:44029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 12:29:11.310550 23120 catalog_manager.cc:4584] TS 4bcc43a92cd5409a8bdb3b73b0c47ae0 (127.7.19.65:44029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3abd4cd5b9446f7b0e1b85868304982: Network error: Client connection negotiation failed: client connection to 127.7.19.65:44029: connect: Connection refused (error 111)
I20260928 12:29:11.314450 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23464
I20260928 12:29:11.319133 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23102
I20260928 12:29:31.324800 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43431
--webserver_interface=127.7.19.126
--webserver_port=44221
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:31.406590 23624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:31.406826 23624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:31.406864 23624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:31.408067 23624 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 12:29:31.408129 23624 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 12:29:31.408150 23624 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 12:29:31.408170 23624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:31.408200 23624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:31.408224 23624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:31.408252 23624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:31.408281 23624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:31.409833 23624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:43431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=44221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:31.410111 23624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:31.410399 23624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:31.415360 23629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:31.415414 23624 server_base.cc:1034] running on GCE node
W20260928 12:29:31.415418 23630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:31.415373 23632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:31.415880 23624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:31.416215 23624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:31.417328 23624 hybrid_clock.cc:648] HybridClock initialized: now 1790598571417314 us; error 32 us; skew 500 ppm
Sep 28 12:29:31 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:31.419955 23624 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:29:31.420456 23624 webserver.cc:460] Webserver started at http://127.7.19.126:44221/ using document root <none> and password file <none>
I20260928 12:29:31.420748 23624 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:31.420819 23624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:31.422187 23624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:31.422873 23639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.423058 23624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:31.423143 23624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "1c3e565e17ea470da03f3eff901e036e"
format_stamp: "Formatted at 2026-09-28 12:28:49 on dist-test-slave-cb7x"
server_key: "e33165cf4e68d557ee065d060597b037"
server_key_iv: "bcf75c243b14c5bf055c0bdbb82c28ac"
server_key_version: "encryptionkey@0"
I20260928 12:29:31.423491 23624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:31.435253 23624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:31.437692 23624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:31.437808 23624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:31.441751 23624 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:43431
I20260928 12:29:31.441805 23691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:43431 every 8 connection(s)
I20260928 12:29:31.442157 23624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:31.443773 23692 sys_catalog.cc:263] Verifying existing consensus state
I20260928 12:29:31.444414 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap starting.
I20260928 12:29:31.449735 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23624
I20260928 12:29:31.450042 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:44029
--local_ip_for_outbound_sockets=127.7.19.65
--tserver_master_addrs=127.7.19.126:43431
--webserver_port=35977
--webserver_interface=127.7.19.65
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:29:31.453361 23692 log.cc:824] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:31.455607 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:29:31.455943 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Bootstrap complete.
I20260928 12:29:31.458258 23692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:31.458474 23692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3e565e17ea470da03f3eff901e036e, State: Initialized, Role: FOLLOWER
I20260928 12:29:31.458657 23692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:31.458719 23692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:31.458796 23692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:31.458841 23692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 2 FOLLOWER]: Advancing to term 3
I20260928 12:29:31.459847 23692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:31.459935 23692 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3e565e17ea470da03f3eff901e036e; no voters:
I20260928 12:29:31.460161 23692 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 12:29:31.460191 23696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 12:29:31.460304 23696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [term 3 LEADER]: Becoming Leader. State: Replica: 1c3e565e17ea470da03f3eff901e036e, State: Running, Role: LEADER
I20260928 12:29:31.460394 23696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } }
I20260928 12:29:31.460424 23692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:31.460714 23697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3e565e17ea470da03f3eff901e036e. Latest consensus state: current_term: 3 leader_uuid: "1c3e565e17ea470da03f3eff901e036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } } }
I20260928 12:29:31.460722 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1c3e565e17ea470da03f3eff901e036e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3e565e17ea470da03f3eff901e036e" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43431 } } }
I20260928 12:29:31.460893 23697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:31.460912 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:31.461477 23705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:31.462268 23705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:31.462364 23705 catalog_manager.cc:1259] Loaded cluster ID: 7dc6660bc3e14c3a813afe99a6314ab8
I20260928 12:29:31.462387 23705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:31.463510 23705 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:31.463693 23705 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Loaded TSK: 0
I20260928 12:29:31.463740 23705 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c3e565e17ea470da03f3eff901e036e: Loaded TSK: 1
F20260928 12:29:31.463960 23705 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbb17dcf9ad google::LogMessage::Fail() at ??:0
@ 0x7fbb17dd288a google::LogMessage::SendToLog() at ??:0
@ 0x7fbb17dcf4de google::LogMessage::Flush() at ??:0
@ 0x7fbb17dd0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbb1a53c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbb1a539d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbb1fd291cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbb1fd2948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbb1fd29594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbb1fcefc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbb1fcf9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbb19c9d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbb19c92954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbb1aaf66db start_thread at ??:0
@ 0x7fbb168cd61f clone at ??:0
W20260928 12:29:31.550756 23694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:31.550966 23694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:31.550992 23694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:31.552176 23694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:31.552218 23694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:31.552246 23694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:31.553762 23694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:44029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=35977
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:31.554004 23694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:31.554251 23694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:31.559324 23733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:31.559324 23736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:31.559510 23694 server_base.cc:1034] running on GCE node
W20260928 12:29:31.559288 23734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:31.559772 23694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:31.560041 23694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:31.561195 23694 hybrid_clock.cc:648] HybridClock initialized: now 1790598571561178 us; error 32 us; skew 500 ppm
Sep 28 12:29:31 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:31.563741 23694 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:29:31.564253 23694 webserver.cc:460] Webserver started at http://127.7.19.65:35977/ using document root <none> and password file <none>
I20260928 12:29:31.564493 23694 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:31.564538 23694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:31.565994 23694 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:31.566769 23743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.566954 23694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:31.567054 23694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "e1095fcc18868a0d33fa17942a704075"
server_key_iv: "f5aecd153d3aa7481bd8997c983e8456"
server_key_version: "encryptionkey@0"
I20260928 12:29:31.567505 23694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:31.586359 23694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:31.589118 23694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:31.589282 23694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:31.589547 23694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:31.590065 23750 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 12:29:31.591540 23694 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 12:29:31.591598 23694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 12:29:31.591632 23694 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 12:29:31.592505 23694 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 12:29:31.592550 23694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 12:29:31.592631 23750 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap starting.
I20260928 12:29:31.594444 23750 log.cc:824] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:31.597690 23750 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 12:29:31.598011 23750 tablet_bootstrap.cc:492] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Bootstrap complete.
I20260928 12:29:31.598163 23750 ts_tablet_manager.cc:1397] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260928 12:29:31.599949 23750 raft_consensus.cc:348] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:31.600309 23694 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:44029
I20260928 12:29:31.600368 23750 raft_consensus.cc:729] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc43a92cd5409a8bdb3b73b0c47ae0, State: Initialized, Role: FOLLOWER
I20260928 12:29:31.600508 23858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:44029 every 8 connection(s)
I20260928 12:29:31.600580 23750 consensus_queue.cc:260] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:31.600747 23694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:31.600899 23750 ts_tablet_manager.cc:1428] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 12:29:31.605949 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23694
I20260928 12:29:31.606242 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:43259
--local_ip_for_outbound_sockets=127.7.19.66
--tserver_master_addrs=127.7.19.126:43431
--webserver_port=41335
--webserver_interface=127.7.19.66
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:31.661623 23862 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:29:31.601420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:45183)
0928 12:29:31.601577 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.601618 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:31.601752 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:31.661413 (+ 59661us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
W20260928 12:29:31.661916 23860 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:43431 (0 consecutive failures): Network error: Failed to ping master at 127.7.19.126:43431: Client connection negotiation failed: client connection to 127.7.19.126:43431: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 12:29:31.662307 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.662166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:51409)
0928 12:29:31.662222 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.662275 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 12:29:31.662657 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.662565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:56811)
0928 12:29:31.662611 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.662638 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:29:31.662791 23860 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:29:31.688246 23863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:31.688439 23863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:31.688462 23863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:31.689675 23863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:31.689723 23863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:31.689749 23863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:31.691174 23863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:43259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=41335
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:31.691408 23863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:31.691646 23863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:31.696516 23868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:31.696523 23871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:31.696528 23869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:31.696853 23863 server_base.cc:1034] running on GCE node
I20260928 12:29:31.697019 23863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:31.697211 23863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:31.698351 23863 hybrid_clock.cc:648] HybridClock initialized: now 1790598571698338 us; error 27 us; skew 500 ppm
Sep 28 12:29:31 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:31.700757 23863 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:29:31.701232 23863 webserver.cc:460] Webserver started at http://127.7.19.66:41335/ using document root <none> and password file <none>
I20260928 12:29:31.701455 23863 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:31.701519 23863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:31.702868 23863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:31.703429 23878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.703585 23863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:31.703685 23863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "3065978401b44167bbfd7e5e55633290"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "93c16668c2e39b3cf7c2a91747521c3b"
server_key_iv: "166bd6caaed5f23a80c33b4e37b02cdb"
server_key_version: "encryptionkey@0"
I20260928 12:29:31.704036 23863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:31.731779 23863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:31.734143 23863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:31.734309 23863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:31.734556 23863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:31.734871 23863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:31.734922 23863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.734957 23863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:31.735006 23863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.740986 23863 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:43259
I20260928 12:29:31.741106 23991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:43259 every 8 connection(s)
I20260928 12:29:31.741423 23863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:29:31.741737 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23863
I20260928 12:29:31.742072 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:34985
--local_ip_for_outbound_sockets=127.7.19.67
--tserver_master_addrs=127.7.19.126:43431
--webserver_port=35861
--webserver_interface=127.7.19.67
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:29:31.742717 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.742256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:57655)
0928 12:29:31.742458 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.742550 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
W20260928 12:29:31.743177 23992 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:43431 (0 consecutive failures): Network error: Failed to ping master at 127.7.19.126:43431: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
I20260928 12:29:31.743494 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.743387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:50109)
0928 12:29:31.743446 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.743475 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:29:31.743813 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.743730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:32867)
0928 12:29:31.743774 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.743797 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:29:31.743937 23992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 12:29:31.823045 23995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:31.823256 23995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:31.823279 23995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:31.824397 23995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:31.824438 23995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:31.824465 23995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:31.825937 23995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=35861
--tserver_master_addrs=127.7.19.126:43431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:31.826172 23995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:31.826395 23995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:31.831418 24003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:31.831454 24000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:31.831414 24001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:31.831702 23995 server_base.cc:1034] running on GCE node
I20260928 12:29:31.831920 23995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:31.832155 23995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:31.833303 23995 hybrid_clock.cc:648] HybridClock initialized: now 1790598571833294 us; error 34 us; skew 500 ppm
Sep 28 12:29:31 dist-test-slave-cb7x krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:31.836103 23995 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:29:31.836632 23995 webserver.cc:460] Webserver started at http://127.7.19.67:35861/ using document root <none> and password file <none>
I20260928 12:29:31.836905 23995 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:31.836952 23995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:31.838261 23995 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:31.838877 24010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.839059 23995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:29:31.839145 23995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "4df5d3bb13fd40afb38b1e06c62aa144"
format_stamp: "Formatted at 2026-09-28 12:28:50 on dist-test-slave-cb7x"
server_key: "3ef24ca1095e2a38ffaf293fb9ae7c3c"
server_key_iv: "ac92ed4d683c0a84782a2721103cad33"
server_key_version: "encryptionkey@0"
I20260928 12:29:31.839457 23995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:31.851701 23995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:31.853847 23995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:31.853955 23995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:31.854135 23995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:31.854460 23995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:31.854494 23995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.854517 23995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:31.854530 23995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:31.860252 23995 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:34985
I20260928 12:29:31.860340 24123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:34985 every 8 connection(s)
I20260928 12:29:31.860692 23995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:29:31.861965 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.861402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:51237)
0928 12:29:31.861562 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.861628 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
W20260928 12:29:31.862218 24124 heartbeater.cc:644] Failed to heartbeat to 127.7.19.126:43431 (0 consecutive failures): Network error: Failed to ping master at 127.7.19.126:43431: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
I20260928 12:29:31.862520 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.862421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:52697)
0928 12:29:31.862478 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.862504 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:29:31.862782 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:31.862720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:53959)
0928 12:29:31.862749 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:31.862768 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 12:29:31.862888 24124 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 12:29:31.867208 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 23995
I20260928 12:29:32.663312 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:32.663093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:51071)
0928 12:29:32.663210 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:32.663254 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 12:29:32.744490 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:32.744271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:54239)
0928 12:29:32.744407 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:32.744445 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260928 12:29:32.863435 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:32.863231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:50429)
0928 12:29:32.863356 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:32.863388 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260928 12:29:33.380699 24131 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:33.380821 24131 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:33.381196 24131 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
Sep 28 12:29:33 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.66@KRBTEST.COM
Sep 28 12:29:33 dist-test-slave-cb7x krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598571, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.67@KRBTEST.COM
I20260928 12:29:33.399133 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.381467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:54491 (local address 127.7.19.66:43259)
0928 12:29:33.381653 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:33.381666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:33.381696 (+ 30us) server_negotiation.cc:407] Connection header received
0928 12:29:33.382202 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:33.382205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:33.382249 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:33.382371 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:33.383195 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.384152 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.385296 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.385670 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.385805 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:33.395260 (+ 9455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:33.395267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:33.395267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:33.395282 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:33.397148 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:33.397685 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:33.397689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:33.397690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:33.397815 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:33.398078 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:33.398079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:33.398080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:33.398437 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:33.398508 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:33.398850 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 12:29:33.399014 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260928 12:29:33.399130 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.381365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:43259 (local address 127.7.19.65:54491)
0928 12:29:33.381462 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:33.381474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:33.381530 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:33.382440 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:33.382443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:33.382846 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:33.383057 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:33.383063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.384254 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.384255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:33.385157 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:33.385160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.385775 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.385776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:33.385879 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:33.393328 (+ 7449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:33.393339 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:33.395059 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:33.397240 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:33.397247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:33.397250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:33.397567 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:33.397896 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:33.397896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:33.397897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:33.398001 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:33.398514 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:33.398521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:33.398659 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 12:29:33.398781 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 12:29:33.398998 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260928 12:29:33.399219 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.381534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:34985 (local address 127.7.19.65:47309)
0928 12:29:33.381616 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:33.381624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:33.381660 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:33.382445 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:33.382447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:33.382843 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:33.383303 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:33.383306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.384454 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.384455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:33.385158 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:33.385161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.385758 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.385759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:33.385882 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:33.393341 (+ 7459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:33.393346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:33.395509 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:33.397812 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:33.397813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:33.397814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:33.398005 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:33.398253 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:33.398254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:33.398254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:33.398313 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:33.398735 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:33.398736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:33.398799 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 12:29:33.398874 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:29:33.399079 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":102,"thread_start_us":51,"threads_started":1}
I20260928 12:29:33.399221 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.381653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:47309 (local address 127.7.19.67:34985)
0928 12:29:33.381810 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:33.381819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:33.381837 (+ 18us) server_negotiation.cc:407] Connection header received
0928 12:29:33.382247 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:33.382249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:33.382284 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:33.382433 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:33.383432 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.384353 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.385244 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:33.385659 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:33.385774 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:33.395678 (+ 9904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:33.395683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:33.395683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:33.395697 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:33.397738 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:33.398107 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:33.398109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:33.398110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:33.398189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:33.398379 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:33.398379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:33.398380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:33.398660 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:33.398749 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:33.398951 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 12:29:33.399094 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
W20260928 12:29:33.400074 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
W20260928 12:29:33.400502 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:33.400566 23745 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc43a92cd5409a8bdb3b73b0c47ae0; no voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144
I20260928 12:29:33.400722 24131 raft_consensus.cc:2726] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:29:33.663924 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.663793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:38035)
0928 12:29:33.663848 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:33.663885 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 12:29:33.745285 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.745048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:36415)
0928 12:29:33.745173 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:33.745217 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260928 12:29:33.864142 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:33.863892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:52595)
0928 12:29:33.864029 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:33.864087 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
I20260928 12:29:34.664629 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:34.664403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:38253)
0928 12:29:34.664544 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:34.664582 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260928 12:29:34.745858 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:34.745634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:33461)
0928 12:29:34.745781 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:34.745818 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260928 12:29:34.865018 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:34.864787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:43105)
0928 12:29:34.864927 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:34.864963 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260928 12:29:35.258531 24141 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:35.258628 24141 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:35.258781 24141 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
W20260928 12:29:35.259317 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
W20260928 12:29:35.259413 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:35.259449 23745 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc43a92cd5409a8bdb3b73b0c47ae0; no voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144
I20260928 12:29:35.259541 24141 raft_consensus.cc:2726] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:29:35.665289 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:35.665071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:55187)
0928 12:29:35.665209 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:35.665245 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 12:29:35.746636 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:35.746364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:53853)
0928 12:29:35.746517 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:35.746562 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260928 12:29:35.865700 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:35.865489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:50975)
0928 12:29:35.865615 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:35.865646 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 12:29:36.666044 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:36.665826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:54343)
0928 12:29:36.665958 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:36.665995 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260928 12:29:36.747318 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:36.747091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:56859)
0928 12:29:36.747240 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:36.747274 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260928 12:29:36.866478 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:36.866245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:52445)
0928 12:29:36.866386 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:36.866438 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260928 12:29:37.388533 24148 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:37.388746 24148 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:37.388957 24148 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
W20260928 12:29:37.389431 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
W20260928 12:29:37.389508 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:37.389523 23745 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc43a92cd5409a8bdb3b73b0c47ae0; no voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144
I20260928 12:29:37.389654 24148 raft_consensus.cc:2726] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:29:37.666883 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:37.666607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:56767)
0928 12:29:37.666791 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:37.666836 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260928 12:29:37.748121 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:37.747840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:47077)
0928 12:29:37.748026 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:37.748071 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260928 12:29:37.867147 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:37.866941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:44759)
0928 12:29:37.867068 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:37.867106 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260928 12:29:38.667629 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:38.667364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:59865)
0928 12:29:38.667527 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:38.667561 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
I20260928 12:29:38.748895 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:38.748630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:40859)
0928 12:29:38.748811 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:38.748850 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
I20260928 12:29:38.867830 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:38.867590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:51991)
0928 12:29:38.867754 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:38.867790 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260928 12:29:39.523082 24155 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:39.523175 24155 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:39.523391 24155 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
W20260928 12:29:39.523855 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
W20260928 12:29:39.523928 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:39.523962 23745 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc43a92cd5409a8bdb3b73b0c47ae0; no voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144
I20260928 12:29:39.524046 24155 raft_consensus.cc:2726] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:29:39.668365 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:39.668146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:54569)
0928 12:29:39.668284 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:39.668320 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260928 12:29:39.749634 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:39.749407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:58127)
0928 12:29:39.749540 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:39.749578 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 12:29:39.868541 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:39.868316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:52861)
0928 12:29:39.868453 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:39.868486 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260928 12:29:40.669328 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:40.669085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:34305)
0928 12:29:40.669242 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:40.669281 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260928 12:29:40.750341 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:40.750097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:60489)
0928 12:29:40.750254 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:40.750293 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":60,"threads_started":1}
I20260928 12:29:40.869186 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:40.868985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:49055)
0928 12:29:40.869106 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:40.869142 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260928 12:29:41.670004 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:41.669800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:60569)
0928 12:29:41.669925 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:41.669958 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260928 12:29:41.751078 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:41.750852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:46335)
0928 12:29:41.750992 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:41.751031 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 12:29:41.869827 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:41.869626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:34337)
0928 12:29:41.869752 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:41.869786 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260928 12:29:42.568166 24165 raft_consensus.cc:482] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:42.568255 24165 raft_consensus.cc:504] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065978401b44167bbfd7e5e55633290" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 43259 } } peers { permanent_uuid: "4df5d3bb13fd40afb38b1e06c62aa144" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 34985 } } peers { permanent_uuid: "4bcc43a92cd5409a8bdb3b73b0c47ae0" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 44029 } }
I20260928 12:29:42.568464 24165 leader_election.cc:288] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259), 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985)
W20260928 12:29:42.568858 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3065978401b44167bbfd7e5e55633290 (127.7.19.66:43259): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
W20260928 12:29:42.568924 23745 leader_election.cc:341] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4df5d3bb13fd40afb38b1e06c62aa144 (127.7.19.67:34985): Not found: Tablet not found: a3abd4cd5b9446f7b0e1b85868304982
I20260928 12:29:42.568940 23745 leader_election.cc:302] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc43a92cd5409a8bdb3b73b0c47ae0; no voters: 3065978401b44167bbfd7e5e55633290, 4df5d3bb13fd40afb38b1e06c62aa144
I20260928 12:29:42.569006 24165 raft_consensus.cc:2726] T a3abd4cd5b9446f7b0e1b85868304982 P 4bcc43a92cd5409a8bdb3b73b0c47ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 12:29:42.670655 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:42.670446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:37371)
0928 12:29:42.670576 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:42.670610 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260928 12:29:42.751868 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:42.751637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:39001)
0928 12:29:42.751779 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:42.751821 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260928 12:29:42.870472 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:42.870250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:43109)
0928 12:29:42.870397 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:42.870431 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260928 12:29:43.671350 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:43.671151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:37627)
0928 12:29:43.671270 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:43.671304 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260928 12:29:43.752547 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:43.752310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:57393)
0928 12:29:43.752454 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:43.752493 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260928 12:29:43.871246 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:43.870972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:34045)
0928 12:29:43.871134 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:43.871192 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260928 12:29:44.672062 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:44.671833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:54049)
0928 12:29:44.671967 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:44.672003 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 12:29:44.753187 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:44.752995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:58363)
0928 12:29:44.753111 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:44.753142 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260928 12:29:44.871886 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:44.871678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:39497)
0928 12:29:44.871815 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:44.871848 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260928 12:29:45.672900 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:45.672538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:50891)
0928 12:29:45.672817 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:45.672854 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":202,"threads_started":1}
I20260928 12:29:45.753913 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:45.753666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:52709)
0928 12:29:45.753826 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:45.753867 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260928 12:29:45.872857 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:45.872331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:52051)
0928 12:29:45.872518 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:45.872556 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260928 12:29:46.673642 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:46.673442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.65:49871)
0928 12:29:46.673564 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:46.673598 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 12:29:46.754575 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:46.754389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.66:39029)
0928 12:29:46.754498 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:46.754529 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260928 12:29:46.869364 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23694
I20260928 12:29:46.873581 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:46.873377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43431 (local address 127.7.19.67:54151)
0928 12:29:46.873501 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:46.873538 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.126:43431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260928 12:29:46.874540 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23863
I20260928 12:29:46.878958 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 23995
2026-09-28T12:29:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58221 ms)
[----------] 30 tests from SecurityITest (229187 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:46 dist-test-slave-cb7x krb5kdc[24183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:46 dist-test-slave-cb7x krb5kdc[24183](info): set up 2 sockets
Sep 28 12:29:46 dist-test-slave-cb7x krb5kdc[24183](info): commencing operation
krb5kdc: starting...
W20260928 12:29:47.932372 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:29:48.047279 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:40285
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:48.126992 24196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:48.127202 24196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:48.127225 24196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:48.128309 24196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:48.128345 24196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:48.128360 24196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:48.128372 24196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:48.128384 24196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:48.129802 24196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:40285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:48.129999 24196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:48.130162 24196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:48.135267 24201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:48.135330 24202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:48.135344 24204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:48.135516 24196 server_base.cc:1034] running on GCE node
I20260928 12:29:48.135720 24196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:48.135965 24196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:48.137102 24196 hybrid_clock.cc:648] HybridClock initialized: now 1790598588137081 us; error 50 us; skew 500 ppm
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:48.139614 24196 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:29:48.140129 24196 webserver.cc:460] Webserver started at http://127.7.19.126:36657/ using document root <none> and password file <none>
I20260928 12:29:48.140383 24196 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:48.140458 24196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:48.140610 24196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:48.141690 24196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "33c643e49cda43d3a5defe52feef753c"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "38d647f3f76a05e52e86fb42c2610c65"
server_key_iv: "5058188fd0cc497f7e21c4c35c869736"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.142055 24196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "33c643e49cda43d3a5defe52feef753c"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "38d647f3f76a05e52e86fb42c2610c65"
server_key_iv: "5058188fd0cc497f7e21c4c35c869736"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.143362 24196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:29:48.144055 24211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.144227 24196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:48.144330 24196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "33c643e49cda43d3a5defe52feef753c"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "38d647f3f76a05e52e86fb42c2610c65"
server_key_iv: "5058188fd0cc497f7e21c4c35c869736"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.144416 24196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:48.163975 24196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:48.166098 24196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:48.166306 24196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:48.170357 24196 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:40285
I20260928 12:29:48.170395 24263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:40285 every 8 connection(s)
I20260928 12:29:48.170759 24196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:48.171594 24264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:48.172098 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24196
I20260928 12:29:48.172185 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:48.172395 7245 external_mini_cluster.cc:1428] Setting key 12fc6dd9dd402fcf04acd168e84b264f
I20260928 12:29:48.174669 24264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c: Bootstrap starting.
I20260928 12:29:48.175299 24264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:48.175711 24264 log.cc:824] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:48.176450 24264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c: No bootstrap required, opened a new log
I20260928 12:29:48.179021 24264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c643e49cda43d3a5defe52feef753c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40285 } }
I20260928 12:29:48.179149 24264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:48.179179 24264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c643e49cda43d3a5defe52feef753c, State: Initialized, Role: FOLLOWER
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:48.179327 24264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c643e49cda43d3a5defe52feef753c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40285 } }
I20260928 12:29:48.179402 24264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:48.179490 24264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:48.179548 24264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:48.180058 24264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c643e49cda43d3a5defe52feef753c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40285 } }
I20260928 12:29:48.180181 24264 leader_election.cc:302] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c643e49cda43d3a5defe52feef753c; no voters:
I20260928 12:29:48.180369 24264 leader_election.cc:288] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:48.180403 24269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:48.180543 24269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [term 1 LEADER]: Becoming Leader. State: Replica: 33c643e49cda43d3a5defe52feef753c, State: Running, Role: LEADER
I20260928 12:29:48.180697 24264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:48.180701 24269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c643e49cda43d3a5defe52feef753c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40285 } }
I20260928 12:29:48.181787 24271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c643e49cda43d3a5defe52feef753c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c643e49cda43d3a5defe52feef753c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40285 } } }
I20260928 12:29:48.181871 24271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:48.182374 24270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c643e49cda43d3a5defe52feef753c. Latest consensus state: current_term: 1 leader_uuid: "33c643e49cda43d3a5defe52feef753c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c643e49cda43d3a5defe52feef753c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40285 } } }
I20260928 12:29:48.182595 24270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:48.182423 24284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:48.183077 24284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:48.183763 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.172945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.7.19.126:40285)
0928 12:29:48.173111 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.173123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.173149 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:29:48.173542 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.173548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.173595 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.173711 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:48.174514 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.175458 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.176552 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.177021 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.177109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.179896 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:48.179899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:48.179900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:48.179911 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:48.182023 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.182455 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.182459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.182459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.182570 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.182791 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.182792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.182793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.183075 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:48.183207 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.183413 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.183555 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 12:29:48.184748 24284 catalog_manager.cc:1347] Generated new cluster ID: 0c435562e8e04178b0d4de0d2660277a
I20260928 12:29:48.184800 24284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:48.193434 24284 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:48.194128 24284 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:48.199514 24284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c: Generated new TSK 0
I20260928 12:29:48.199678 24284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:29:48.236722 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40285
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:48.336194 24290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:48.336404 24290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:48.336427 24290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:48.337695 24290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:48.337746 24290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:48.337774 24290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:48.339223 24290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:48.339465 24290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:48.339691 24290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:48.344784 24296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:48.344810 24295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:48.344921 24290 server_base.cc:1034] running on GCE node
W20260928 12:29:48.344928 24298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:48.345295 24290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:48.345501 24290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:48.346683 24290 hybrid_clock.cc:648] HybridClock initialized: now 1790598588346665 us; error 31 us; skew 500 ppm
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:48.349392 24290 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:29:48.349879 24290 webserver.cc:460] Webserver started at http://127.7.19.65:34653/ using document root <none> and password file <none>
I20260928 12:29:48.350124 24290 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:48.350195 24290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:48.350348 24290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:48.351416 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "282ef73523b1481db7acce12c9f0f198"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "819eb6f3beeb1f33501c6b3dcb38a356"
server_key_iv: "c6cce21902ca16caed4c98dbea216a0e"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.351794 24290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "282ef73523b1481db7acce12c9f0f198"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "819eb6f3beeb1f33501c6b3dcb38a356"
server_key_iv: "c6cce21902ca16caed4c98dbea216a0e"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.353164 24290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:48.353859 24305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.354032 24290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:48.354132 24290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "282ef73523b1481db7acce12c9f0f198"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "819eb6f3beeb1f33501c6b3dcb38a356"
server_key_iv: "c6cce21902ca16caed4c98dbea216a0e"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.354211 24290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:48.369776 24290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:48.371994 24290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:48.372159 24290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:48.372390 24290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:48.372793 24290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:48.372851 24290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.372891 24290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:48.372918 24290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.378602 24290 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:37519
I20260928 12:29:48.378650 24418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:37519 every 8 connection(s)
I20260928 12:29:48.378969 24290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:48.381711 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24290
I20260928 12:29:48.381847 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:29:48.382062 7245 external_mini_cluster.cc:1428] Setting key abb49cd994c135197a364117e112897c
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:48.389726 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.379720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:50149 (local address 127.7.19.126:40285)
0928 12:29:48.379780 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.379782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.379991 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:29:48.380546 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.380547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.380560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.380619 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:48.381836 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.382604 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.384149 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.384366 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.384454 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.386905 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:48.386907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:48.386907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:48.386914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:48.388281 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.388737 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.388738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.388738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.388802 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.389060 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.389061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.389062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.389183 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:48.389263 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.389538 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.389623 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 12:29:48.389896 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.379694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40285 (local address 127.7.19.65:50149)
0928 12:29:48.379874 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.379910 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.380432 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.380719 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.380723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.381390 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:48.381653 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.381660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.382732 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.382733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.384041 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.384044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.384464 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.384466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.384605 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.385233 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:48.385242 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:48.386799 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:48.388382 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:48.388389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:48.388391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:48.388608 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:48.388873 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:48.388874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:48.388875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:48.388991 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:48.389254 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:48.389260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:48.389402 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.389506 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.389664 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
I20260928 12:29:48.390344 24419 heartbeater.cc:347] Connected to a master server at 127.7.19.126:40285
I20260928 12:29:48.390442 24419 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:48.390645 24419 heartbeater.cc:511] Master 127.7.19.126:40285 requested a full tablet report, sending...
I20260928 12:29:48.391157 24228 ts_manager.cc:194] Registered new tserver with Master: 282ef73523b1481db7acce12c9f0f198 (127.7.19.65:37519)
I20260928 12:29:48.391767 24228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:50149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:29:48.408080 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40285
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:48.490077 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:48.490316 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:48.490353 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:48.491504 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:48.491572 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:48.491618 24424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:48.493144 24424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:48.493402 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:48.493633 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:48.498772 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:48.498931 24424 server_base.cc:1034] running on GCE node
W20260928 12:29:48.498786 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:48.498778 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:48.499213 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:48.499477 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:48.500732 24424 hybrid_clock.cc:648] HybridClock initialized: now 1790598588500711 us; error 27 us; skew 500 ppm
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:48.503261 24424 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:29:48.503681 24424 webserver.cc:460] Webserver started at http://127.7.19.66:45529/ using document root <none> and password file <none>
I20260928 12:29:48.503918 24424 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:48.503988 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:48.504136 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:48.505241 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "8f68ed069f604f90a4ce537e519219a7"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "14cac09cf5329a3de36716ea58ca25cd"
server_key_iv: "6519b2598de0b7f832575c223cdf8697"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.505604 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "8f68ed069f604f90a4ce537e519219a7"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "14cac09cf5329a3de36716ea58ca25cd"
server_key_iv: "6519b2598de0b7f832575c223cdf8697"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.506920 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:29:48.507617 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.507776 24424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:48.507864 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "8f68ed069f604f90a4ce537e519219a7"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "14cac09cf5329a3de36716ea58ca25cd"
server_key_iv: "6519b2598de0b7f832575c223cdf8697"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.507943 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:48.524564 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:48.526659 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:48.526837 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:48.527099 24424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:48.527448 24424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:48.527513 24424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.527563 24424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:48.527606 24424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.533380 24424 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:42149
I20260928 12:29:48.533443 24552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:42149 every 8 connection(s)
I20260928 12:29:48.533797 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:48.543033 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24424
I20260928 12:29:48.543150 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:29:48.543170 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.534648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:41529 (local address 127.7.19.126:40285)
0928 12:29:48.534727 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.534729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.534980 (+ 251us) server_negotiation.cc:407] Connection header received
0928 12:29:48.535468 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.535470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.535480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.535549 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:48.536478 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.537011 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.538012 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.538217 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.538281 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.540383 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:48.540384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:48.540384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:48.540391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:48.541912 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.542271 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.542272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.542272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.542323 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.542528 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.542529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.542529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.542642 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:48.542698 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.542947 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.543057 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:29:48.543287 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.534645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40285 (local address 127.7.19.66:41529)
0928 12:29:48.534848 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.534887 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.535372 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.535602 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.535605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.536177 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:48.536382 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.536387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.537102 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.537103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.537917 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.537919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.538297 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.538298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.538398 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.538891 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:48.538899 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:48.540296 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:48.541985 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:48.541990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:48.541992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:48.542203 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:48.542381 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:48.542382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:48.542382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:48.542472 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:48.542702 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:48.542706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:48.542817 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.542898 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.543024 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
I20260928 12:29:48.543378 7245 external_mini_cluster.cc:1428] Setting key 3ee0eab6df18b017c94d3cc072e00fe7
I20260928 12:29:48.543869 24553 heartbeater.cc:347] Connected to a master server at 127.7.19.126:40285
I20260928 12:29:48.543985 24553 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:48.544206 24553 heartbeater.cc:511] Master 127.7.19.126:40285 requested a full tablet report, sending...
I20260928 12:29:48.544627 24228 ts_manager.cc:194] Registered new tserver with Master: 8f68ed069f604f90a4ce537e519219a7 (127.7.19.66:42149)
I20260928 12:29:48.545034 24228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:41529
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:29:48.567689 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40285
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:48.648207 24558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:48.648416 24558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:48.648440 24558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:48.649601 24558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:48.649645 24558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:48.649705 24558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:48.651257 24558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:40285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:48.651511 24558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:48.651758 24558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:48.656782 24564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:48.656925 24563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:48.657007 24566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:48.656867 24558 server_base.cc:1034] running on GCE node
I20260928 12:29:48.657272 24558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:48.657532 24558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:48.658695 24558 hybrid_clock.cc:648] HybridClock initialized: now 1790598588658685 us; error 39 us; skew 500 ppm
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:48.661302 24558 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:29:48.661665 24558 webserver.cc:460] Webserver started at http://127.7.19.67:37741/ using document root <none> and password file <none>
I20260928 12:29:48.661860 24558 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:48.661903 24558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:48.662040 24558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:48.663077 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "045e65bc01e44718a408684d12d725d1"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "ff155ecb788e720e0c88b2d223d26365"
server_key_iv: "e422143d47188357ff7fec3707136b71"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.663462 24558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "045e65bc01e44718a408684d12d725d1"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "ff155ecb788e720e0c88b2d223d26365"
server_key_iv: "e422143d47188357ff7fec3707136b71"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.664794 24558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:48.665534 24573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.665699 24558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:48.665802 24558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "045e65bc01e44718a408684d12d725d1"
format_stamp: "Formatted at 2026-09-28 12:29:48 on dist-test-slave-cb7x"
server_key: "ff155ecb788e720e0c88b2d223d26365"
server_key_iv: "e422143d47188357ff7fec3707136b71"
server_key_version: "encryptionkey@0"
I20260928 12:29:48.665874 24558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:48.677996 24558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:48.680284 24558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:48.680496 24558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:48.680831 24558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:48.681150 24558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:48.681216 24558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.681260 24558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:48.681290 24558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:48.687440 24558 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:40181
I20260928 12:29:48.687570 24686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:40181 every 8 connection(s)
I20260928 12:29:48.687849 24558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:29:48.692418 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24558
I20260928 12:29:48.692515 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:29:48.692734 7245 external_mini_cluster.cc:1428] Setting key d53f74e152a4582426a298f809f8494f
Sep 28 12:29:48 dist-test-slave-cb7x krb5kdc[24183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598588, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:48.696857 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.688531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:49133 (local address 127.7.19.126:40285)
0928 12:29:48.688573 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.688575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.688807 (+ 232us) server_negotiation.cc:407] Connection header received
0928 12:29:48.689307 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.689308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.689317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.689379 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:48.690250 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.690748 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.691720 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.691898 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.691950 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.694219 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:48.694221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:48.694221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:48.694227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:48.695627 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.696008 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.696009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.696009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.696062 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.696287 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.696287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.696288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.696388 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:48.696439 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.696717 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.696795 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:29:48.697021 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.688498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:40285 (local address 127.7.19.67:49133)
0928 12:29:48.688665 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.688730 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.689220 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.689434 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.689437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.689977 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:48.690174 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.690179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.690832 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.690833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.691639 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.691642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.691967 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.691968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.692062 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.692653 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:48.692661 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:48.694133 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:48.695710 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:48.695715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:48.695716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:48.695928 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:48.696122 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:48.696122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:48.696123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:48.696217 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:48.696444 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:48.696448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:48.696556 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.696638 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.696827 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":143,"threads_started":1}
I20260928 12:29:48.697557 24687 heartbeater.cc:347] Connected to a master server at 127.7.19.126:40285
I20260928 12:29:48.697655 24687 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:48.697829 24687 heartbeater.cc:511] Master 127.7.19.126:40285 requested a full tablet report, sending...
I20260928 12:29:48.698199 24228 ts_manager.cc:194] Registered new tserver with Master: 045e65bc01e44718a408684d12d725d1 (127.7.19.67:40181)
I20260928 12:29:48.698522 24228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:49133
I20260928 12:29:48.704438 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:48.711808 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.705768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.7.19.126:40285)
0928 12:29:48.705837 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.705839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.705916 (+ 77us) server_negotiation.cc:407] Connection header received
0928 12:29:48.706003 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.706004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.706014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.706069 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:48.706611 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.707087 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.707840 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.708047 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.708124 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.709384 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:48.709386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:48.709386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:48.709394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:48.710636 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.710987 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.710989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.710989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.711045 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:48.711255 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:48.711256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:48.711257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:48.711408 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:48.711515 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.711630 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.711706 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 12:29:48.713585 24228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:48.715252 24228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:48.724030 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.718468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:37519 (local address 127.0.0.1:45810)
0928 12:29:48.719322 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.719361 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.719437 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.719860 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.719861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.719862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:48.719883 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.719884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.721266 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.721267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.722586 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.722589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.723612 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.723613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.723766 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.723788 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.723856 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.723918 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":65,"threads_started":1}
I20260928 12:29:48.724165 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.718574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.7.19.65:37519)
0928 12:29:48.718907 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.718912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.719530 (+ 618us) server_negotiation.cc:407] Connection header received
0928 12:29:48.719572 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.719575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.719594 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.719668 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:48.719971 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.721144 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.723020 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.723513 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.723601 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.723667 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.724021 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.724073 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":90,"threads_started":1}
I20260928 12:29:48.724169 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.718972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.7.19.66:42149)
0928 12:29:48.719202 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.719207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.719943 (+ 736us) server_negotiation.cc:407] Connection header received
0928 12:29:48.719995 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.719998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.720015 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.720102 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:48.720992 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.721948 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.722990 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.723716 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.723789 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.723842 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.724036 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.724088 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260928 12:29:48.724396 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.719186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:42149 (local address 127.0.0.1:55742)
0928 12:29:48.719848 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.719859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.719923 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.720109 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.720110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.720112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:48.720887 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.720891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.722043 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.722044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.722925 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.722928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.723877 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.723879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.723946 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.723952 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.724286 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.724328 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":68,"threads_started":1}
I20260928 12:29:48.725083 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.718604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40181 (local address 127.0.0.1:59050)
0928 12:29:48.719393 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.719402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.719452 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.719676 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.719707 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.719710 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:48.719792 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.719794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.722675 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.722676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.723806 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.723809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.724870 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.724871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.724931 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.724934 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.724983 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.725019 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":101,"threads_started":1}
I20260928 12:29:48.725137 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.718658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.7.19.67:40181)
0928 12:29:48.718851 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.718856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.719455 (+ 599us) server_negotiation.cc:407] Connection header received
0928 12:29:48.719545 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.719548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.719585 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.719674 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:48.720247 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.721333 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.723908 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.724716 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.724787 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.724838 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.725015 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.725060 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260928 12:29:48.725205 24353 tablet_service.cc:1467] Processing CreateTablet for tablet 79482a04bea64ef6aa60885d10e85e62 (DEFAULT_TABLE table=test-table [id=36f2210283ee4fc3bd4e2f43a0b3dbf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:48.725307 24487 tablet_service.cc:1467] Processing CreateTablet for tablet 79482a04bea64ef6aa60885d10e85e62 (DEFAULT_TABLE table=test-table [id=36f2210283ee4fc3bd4e2f43a0b3dbf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:48.725576 24353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79482a04bea64ef6aa60885d10e85e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:48.725621 24487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79482a04bea64ef6aa60885d10e85e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:48.728223 24704 tablet_bootstrap.cc:492] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: Bootstrap starting.
I20260928 12:29:48.728221 24705 tablet_bootstrap.cc:492] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Bootstrap starting.
I20260928 12:29:48.728910 24704 tablet_bootstrap.cc:654] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:48.728940 24705 tablet_bootstrap.cc:654] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:48.728967 24621 tablet_service.cc:1467] Processing CreateTablet for tablet 79482a04bea64ef6aa60885d10e85e62 (DEFAULT_TABLE table=test-table [id=36f2210283ee4fc3bd4e2f43a0b3dbf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:48.729336 24621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79482a04bea64ef6aa60885d10e85e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:48.729512 24704 log.cc:824] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:48.729518 24705 log.cc:824] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:48.730321 24705 tablet_bootstrap.cc:492] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: No bootstrap required, opened a new log
I20260928 12:29:48.730433 24705 ts_tablet_manager.cc:1397] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:48.730511 24704 tablet_bootstrap.cc:492] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: No bootstrap required, opened a new log
I20260928 12:29:48.730589 24704 ts_tablet_manager.cc:1397] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:48.731911 24704 raft_consensus.cc:348] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.732081 24704 raft_consensus.cc:374] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:48.732143 24704 raft_consensus.cc:729] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282ef73523b1481db7acce12c9f0f198, State: Initialized, Role: FOLLOWER
I20260928 12:29:48.732270 24704 consensus_queue.cc:260] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.732550 24704 ts_tablet_manager.cc:1428] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:48.732501 24705 raft_consensus.cc:348] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.732729 24419 heartbeater.cc:503] Master 127.7.19.126:40285 was elected leader, sending a full tablet report...
I20260928 12:29:48.732782 24709 tablet_bootstrap.cc:492] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Bootstrap starting.
I20260928 12:29:48.732805 24705 raft_consensus.cc:374] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:48.732921 24705 raft_consensus.cc:729] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f68ed069f604f90a4ce537e519219a7, State: Initialized, Role: FOLLOWER
I20260928 12:29:48.733088 24705 consensus_queue.cc:260] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.733387 24553 heartbeater.cc:503] Master 127.7.19.126:40285 was elected leader, sending a full tablet report...
I20260928 12:29:48.733369 24705 ts_tablet_manager.cc:1428] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:29:48.733489 24709 tablet_bootstrap.cc:654] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:48.734022 24709 log.cc:824] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:48.734773 24709 tablet_bootstrap.cc:492] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: No bootstrap required, opened a new log
I20260928 12:29:48.734858 24709 ts_tablet_manager.cc:1397] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:29:48.736155 24709 raft_consensus.cc:348] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.736310 24709 raft_consensus.cc:374] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:48.736356 24709 raft_consensus.cc:729] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045e65bc01e44718a408684d12d725d1, State: Initialized, Role: FOLLOWER
I20260928 12:29:48.736464 24709 consensus_queue.cc:260] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.736800 24709 ts_tablet_manager.cc:1428] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 12:29:48.736863 24687 heartbeater.cc:503] Master 127.7.19.126:40285 was elected leader, sending a full tablet report...
W20260928 12:29:48.784375 24554 tablet.cc:2369] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:29:48.831831 24710 raft_consensus.cc:482] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:48.832029 24710 raft_consensus.cc:504] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.832442 24710 leader_election.cc:288] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282ef73523b1481db7acce12c9f0f198 (127.7.19.65:37519), 045e65bc01e44718a408684d12d725d1 (127.7.19.67:40181)
I20260928 12:29:48.835584 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.832781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:48209 (local address 127.7.19.67:40181)
0928 12:29:48.832829 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.832831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.832976 (+ 145us) server_negotiation.cc:407] Connection header received
0928 12:29:48.833045 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.833045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.833057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.833128 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:48.833283 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.833734 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.834859 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.835228 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.835320 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.835330 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.835483 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.835518 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 12:29:48.835585 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.832732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:55885 (local address 127.7.19.65:37519)
0928 12:29:48.832777 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.832779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.832843 (+ 64us) server_negotiation.cc:407] Connection header received
0928 12:29:48.832936 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.832937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.832949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.833011 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:48.833210 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.833689 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.834851 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.835237 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.835320 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.835331 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.835479 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.835516 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:29:48.835593 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.832631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:37519 (local address 127.7.19.66:55885)
0928 12:29:48.832782 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.832794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.832858 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.833035 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.833035 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.833038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:48.833114 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.833116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.833797 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.833798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.834755 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.834757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.835318 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.835319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.835399 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.835402 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.835473 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.835512 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":22}
I20260928 12:29:48.835598 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.832634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40181 (local address 127.7.19.66:48209)
0928 12:29:48.832898 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.832909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:48.832982 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:48.833132 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:48.833133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:48.833135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:48.833210 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.833212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.833823 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.833823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.834769 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:48.834771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.835321 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.835322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:48.835393 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.835397 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:48.835467 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:29:48.835505 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":74,"thread_start_us":116,"threads_started":1}
I20260928 12:29:48.835973 24373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79482a04bea64ef6aa60885d10e85e62" candidate_uuid: "8f68ed069f604f90a4ce537e519219a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282ef73523b1481db7acce12c9f0f198" is_pre_election: true
I20260928 12:29:48.835969 24641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79482a04bea64ef6aa60885d10e85e62" candidate_uuid: "8f68ed069f604f90a4ce537e519219a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045e65bc01e44718a408684d12d725d1" is_pre_election: true
I20260928 12:29:48.836119 24641 raft_consensus.cc:2454] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f68ed069f604f90a4ce537e519219a7 in term 0.
I20260928 12:29:48.836119 24373 raft_consensus.cc:2454] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f68ed069f604f90a4ce537e519219a7 in term 0.
I20260928 12:29:48.836347 24440 leader_election.cc:302] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045e65bc01e44718a408684d12d725d1, 8f68ed069f604f90a4ce537e519219a7; no voters:
I20260928 12:29:48.836515 24710 raft_consensus.cc:2781] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:48.836575 24710 raft_consensus.cc:482] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:48.836597 24710 raft_consensus.cc:3037] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:48.837308 24710 raft_consensus.cc:504] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.837445 24710 leader_election.cc:288] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [CANDIDATE]: Term 1 election: Requested vote from peers 282ef73523b1481db7acce12c9f0f198 (127.7.19.65:37519), 045e65bc01e44718a408684d12d725d1 (127.7.19.67:40181)
I20260928 12:29:48.837610 24641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79482a04bea64ef6aa60885d10e85e62" candidate_uuid: "8f68ed069f604f90a4ce537e519219a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045e65bc01e44718a408684d12d725d1"
I20260928 12:29:48.837625 24373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79482a04bea64ef6aa60885d10e85e62" candidate_uuid: "8f68ed069f604f90a4ce537e519219a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282ef73523b1481db7acce12c9f0f198"
I20260928 12:29:48.837700 24373 raft_consensus.cc:3037] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:48.837700 24641 raft_consensus.cc:3037] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:48.838625 24373 raft_consensus.cc:2454] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f68ed069f604f90a4ce537e519219a7 in term 1.
I20260928 12:29:48.838632 24641 raft_consensus.cc:2454] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f68ed069f604f90a4ce537e519219a7 in term 1.
I20260928 12:29:48.838830 24440 leader_election.cc:302] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045e65bc01e44718a408684d12d725d1, 8f68ed069f604f90a4ce537e519219a7; no voters:
I20260928 12:29:48.838932 24710 raft_consensus.cc:2781] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:48.839075 24710 raft_consensus.cc:686] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 1 LEADER]: Becoming Leader. State: Replica: 8f68ed069f604f90a4ce537e519219a7, State: Running, Role: LEADER
I20260928 12:29:48.839170 24710 consensus_queue.cc:237] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } }
I20260928 12:29:48.840045 24227 catalog_manager.cc:5495] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f68ed069f604f90a4ce537e519219a7 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "8f68ed069f604f90a4ce537e519219a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f68ed069f604f90a4ce537e519219a7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 42149 } health_report { overall_health: HEALTHY } } }
I20260928 12:29:48.867887 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.865335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.7.19.66:42149)
0928 12:29:48.865401 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:48.865403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:48.865413 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:29:48.865499 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:48.865500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:48.865520 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:48.865602 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:48.865742 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.866218 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.867030 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:48.867241 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:48.867329 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:48.867503 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:48.867614 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:48.867708 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:48.867731 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 12:29:48.867752 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:29:48.869849 24641 raft_consensus.cc:1261] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 1 FOLLOWER]: Refusing update from remote peer 8f68ed069f604f90a4ce537e519219a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:48.869849 24373 raft_consensus.cc:1261] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 1 FOLLOWER]: Refusing update from remote peer 8f68ed069f604f90a4ce537e519219a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:48.870276 24714 consensus_queue.cc:1035] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "045e65bc01e44718a408684d12d725d1" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:48.870429 24710 consensus_queue.cc:1035] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282ef73523b1481db7acce12c9f0f198" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:48.872850 24717 mvcc.cc:205] Tried to move back new op lower bound from 7334291820007952384 to 7334291819885834240. Current Snapshot: MvccSnapshot[applied={T|T < 7334291820007952384}]
I20260928 12:29:48.873438 24718 mvcc.cc:205] Tried to move back new op lower bound from 7334291820007952384 to 7334291819885834240. Current Snapshot: MvccSnapshot[applied={T|T < 7334291820007952384}]
I20260928 12:29:48.873483 24719 mvcc.cc:205] Tried to move back new op lower bound from 7334291820007952384 to 7334291819885834240. Current Snapshot: MvccSnapshot[applied={T|T < 7334291820007952384}]
I20260928 12:29:48.875715 24228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:48.875820 24228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:48.876607 24228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c643e49cda43d3a5defe52feef753c: Sending DeleteTablet for 3 replicas of tablet 79482a04bea64ef6aa60885d10e85e62
I20260928 12:29:48.876964 24353 tablet_service.cc:1514] Processing DeleteTablet for tablet 79482a04bea64ef6aa60885d10e85e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:48 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:45810
I20260928 12:29:48.876981 24487 tablet_service.cc:1514] Processing DeleteTablet for tablet 79482a04bea64ef6aa60885d10e85e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:48 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:55742
I20260928 12:29:48.877007 24621 tablet_service.cc:1514] Processing DeleteTablet for tablet 79482a04bea64ef6aa60885d10e85e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:48 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:59050
I20260928 12:29:48.877204 24727 tablet_replica.cc:330] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198: stopping tablet replica
I20260928 12:29:48.877296 24727 raft_consensus.cc:2229] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:48.877378 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24290
I20260928 12:29:48.877396 24727 raft_consensus.cc:2258] T 79482a04bea64ef6aa60885d10e85e62 P 282ef73523b1481db7acce12c9f0f198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:48.878744 24729 tablet_replica.cc:330] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: stopping tablet replica
I20260928 12:29:48.878856 24729 raft_consensus.cc:2229] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:48.878955 24729 raft_consensus.cc:2258] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:48.879343 24729 ts_tablet_manager.cc:1905] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:48.880860 24729 ts_tablet_manager.cc:1918] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:48.880990 24729 log.cc:1196] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/79482a04bea64ef6aa60885d10e85e62
I20260928 12:29:48.881258 24729 ts_tablet_manager.cc:1939] T 79482a04bea64ef6aa60885d10e85e62 P 045e65bc01e44718a408684d12d725d1: Deleting consensus metadata
I20260928 12:29:48.881652 24212 catalog_manager.cc:4841] TS 045e65bc01e44718a408684d12d725d1 (127.7.19.67:40181): tablet 79482a04bea64ef6aa60885d10e85e62 (table test-table [id=36f2210283ee4fc3bd4e2f43a0b3dbf8]) successfully deleted
W20260928 12:29:48.884660 24213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:37519 (error 108)
I20260928 12:29:48.884832 24728 tablet_replica.cc:330] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: stopping tablet replica
I20260928 12:29:48.884961 24728 raft_consensus.cc:2229] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:29:48.885104 24728 raft_consensus.cc:2258] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:48.885322 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.885177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:37519 (local address 127.0.0.1:45818)
0928 12:29:48.885229 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.885270 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:37519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:29:48.885471 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24424
I20260928 12:29:48.885596 24728 ts_tablet_manager.cc:1905] T 79482a04bea64ef6aa60885d10e85e62 P 8f68ed069f604f90a4ce537e519219a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 12:29:48.886622 24213 catalog_manager.cc:4584] TS 282ef73523b1481db7acce12c9f0f198 (127.7.19.65:37519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79482a04bea64ef6aa60885d10e85e62: Network error: Client connection negotiation failed: client connection to 127.7.19.65:37519: connect: Connection refused (error 111)
I20260928 12:29:48.890664 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24558
I20260928 12:29:48.890748 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:48.890616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:42149 (local address 127.0.0.1:55764)
0928 12:29:48.890680 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:48.890714 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.66:42149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 12:29:48.895828 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24196
2026-09-28T12:29:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2017 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T12:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:48Z Disabled control of system clock
I20260928 12:29:48.923447 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39185
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:49.003818 24734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:49.004055 24734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:49.004076 24734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:49.005350 24734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:49.005394 24734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:49.005407 24734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:49.005420 24734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:49.005432 24734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:49.006836 24734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:39185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:49.007098 24734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:49.007406 24734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:49.012257 24740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.012266 24742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:49.012368 24734 server_base.cc:1034] running on GCE node
W20260928 12:29:49.012266 24739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:49.012709 24734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:49.013011 24734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:49.014191 24734 hybrid_clock.cc:648] HybridClock initialized: now 1790598589014177 us; error 26 us; skew 500 ppm
I20260928 12:29:49.015714 24734 webserver.cc:460] Webserver started at http://127.7.19.126:32793/ using document root <none> and password file <none>
I20260928 12:29:49.015971 24734 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:49.016047 24734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:49.016206 24734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:49.017402 24734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "19e552c4332342378ab4cee295d3dff1"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "a62020b50de82f7b361229a056a28573"
server_key_iv: "bdb0e504fdff854fefbf38ff6317c56a"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.017802 24734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "19e552c4332342378ab4cee295d3dff1"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "a62020b50de82f7b361229a056a28573"
server_key_iv: "bdb0e504fdff854fefbf38ff6317c56a"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.019096 24734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:29:49.019882 24748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.020076 24734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:49.020195 24734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "19e552c4332342378ab4cee295d3dff1"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "a62020b50de82f7b361229a056a28573"
server_key_iv: "bdb0e504fdff854fefbf38ff6317c56a"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.020277 24734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:49.036154 24734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:49.036481 24734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:49.036643 24734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:49.040510 24734 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:39185
I20260928 12:29:49.040545 24800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:39185 every 8 connection(s)
I20260928 12:29:49.040942 24734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:49.041999 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:49.044246 24801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1: Bootstrap starting.
I20260928 12:29:49.044828 24801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:49.045121 24801 log.cc:824] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:49.045761 24801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1: No bootstrap required, opened a new log
I20260928 12:29:49.047317 24801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e552c4332342378ab4cee295d3dff1" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39185 } }
I20260928 12:29:49.047451 24801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:49.047474 24801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e552c4332342378ab4cee295d3dff1, State: Initialized, Role: FOLLOWER
I20260928 12:29:49.047654 24801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e552c4332342378ab4cee295d3dff1" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39185 } }
I20260928 12:29:49.047721 24801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:49.047744 24801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:49.047752 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24734
I20260928 12:29:49.047770 24801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:49.047874 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:49.048084 7245 external_mini_cluster.cc:1428] Setting key 8c0a0a9f27c205511c38038a7c88af59
I20260928 12:29:49.048454 24801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e552c4332342378ab4cee295d3dff1" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39185 } }
I20260928 12:29:49.048604 24801 leader_election.cc:302] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19e552c4332342378ab4cee295d3dff1; no voters:
I20260928 12:29:49.048808 24801 leader_election.cc:288] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:49.048849 24805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:49.048988 24805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [term 1 LEADER]: Becoming Leader. State: Replica: 19e552c4332342378ab4cee295d3dff1, State: Running, Role: LEADER
I20260928 12:29:49.049120 24801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:49.049115 24805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e552c4332342378ab4cee295d3dff1" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39185 } }
I20260928 12:29:49.049801 24808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19e552c4332342378ab4cee295d3dff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e552c4332342378ab4cee295d3dff1" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39185 } } }
I20260928 12:29:49.049901 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19e552c4332342378ab4cee295d3dff1. Latest consensus state: current_term: 1 leader_uuid: "19e552c4332342378ab4cee295d3dff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e552c4332342378ab4cee295d3dff1" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39185 } } }
I20260928 12:29:49.049960 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:49.049911 24808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1 [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:49.050213 24815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:49.050756 24815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:49.052421 24815 catalog_manager.cc:1347] Generated new cluster ID: e37d4862e975458d8db89d4d3f4cceb2
I20260928 12:29:49.052477 24815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:49.054561 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.048642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.7.19.126:39185)
0928 12:29:49.048922 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.048930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.048962 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:29:49.049073 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.049080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.049151 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.049370 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:49.049642 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.051577 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.052593 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.053060 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.053200 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.053927 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:49.053931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:49.053931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:49.054102 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:49.054133 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:49.054143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:49.054214 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.054316 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.054395 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":139,"threads_started":1}
I20260928 12:29:49.068136 24815 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:49.068662 24815 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:49.073537 24815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1: Generated new TSK 0
I20260928 12:29:49.073714 24815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:29:49.080212 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39185
--builtin_ntp_servers=127.7.19.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:49.160495 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:49.160737 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:49.160773 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:49.161887 24825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:49.161948 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:49.161990 24825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:49.163303 24825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:49.163555 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:49.163836 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:49.170158 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.170178 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.170385 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:49.170284 24825 server_base.cc:1034] running on GCE node
I20260928 12:29:49.170722 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:49.171011 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:49.172204 24825 hybrid_clock.cc:648] HybridClock initialized: now 1790598589172188 us; error 42 us; skew 500 ppm
I20260928 12:29:49.173709 24825 webserver.cc:460] Webserver started at http://127.7.19.65:46591/ using document root <none> and password file <none>
I20260928 12:29:49.173985 24825 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:49.174045 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:49.174180 24825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:49.175613 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "f828485f1bf34269b0b7e550be166578"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "34dbb094931fe006f1532d3828c4f5dc"
server_key_iv: "a67f8981e54dc91424b447fb318a8848"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.176031 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "f828485f1bf34269b0b7e550be166578"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "34dbb094931fe006f1532d3828c4f5dc"
server_key_iv: "a67f8981e54dc91424b447fb318a8848"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.177723 24825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 12:29:49.178637 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.178835 24825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:49.178936 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "f828485f1bf34269b0b7e550be166578"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "34dbb094931fe006f1532d3828c4f5dc"
server_key_iv: "a67f8981e54dc91424b447fb318a8848"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.179015 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:49.192420 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:49.192746 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:49.192894 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:49.193109 24825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:49.193414 24825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:49.193449 24825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.193468 24825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:49.193513 24825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.199469 24825 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:42555
I20260928 12:29:49.199505 24952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:42555 every 8 connection(s)
I20260928 12:29:49.199824 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:49.204867 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.200512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:53009 (local address 127.7.19.126:39185)
0928 12:29:49.200609 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.200611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.200904 (+ 293us) server_negotiation.cc:407] Connection header received
0928 12:29:49.201367 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.201368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.201380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.201431 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:49.202041 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.202530 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.203512 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.203726 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.203794 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.204516 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:49.204517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:49.204518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:49.204524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:49.204542 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:49.204546 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:49.204603 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.204733 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.204764 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 12:29:49.204928 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24825
I20260928 12:29:49.205000 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.200493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39185 (local address 127.7.19.65:53009)
0928 12:29:49.200789 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.200822 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.201278 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.201480 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.201483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.201491 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:49.201951 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.201954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.202631 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.202632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.203412 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.203414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.203808 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.203809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.203912 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.204384 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:49.204385 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:49.204411 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:49.204431 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:49.204609 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:49.204613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:49.204615 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.204712 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.204823 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":157,"threads_started":1}
I20260928 12:29:49.205029 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:29:49.205236 7245 external_mini_cluster.cc:1428] Setting key 1ef19abeb935ca2cdb79071202eedff6
I20260928 12:29:49.205546 24953 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39185
I20260928 12:29:49.205708 24953 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:49.205899 24953 heartbeater.cc:511] Master 127.7.19.126:39185 requested a full tablet report, sending...
I20260928 12:29:49.206219 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39185
--builtin_ntp_servers=127.7.19.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:29:49.206393 24765 ts_manager.cc:194] Registered new tserver with Master: f828485f1bf34269b0b7e550be166578 (127.7.19.65:42555)
I20260928 12:29:49.206964 24765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.65:53009
W20260928 12:29:49.286134 24956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:49.286342 24956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:49.286365 24956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:49.287623 24956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:49.287663 24956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:49.287688 24956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:49.289062 24956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:49.289312 24956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:49.289610 24956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:49.294765 24964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.294742 24962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.294754 24961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:49.294848 24956 server_base.cc:1034] running on GCE node
I20260928 12:29:49.295219 24956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:49.295418 24956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:49.296591 24956 hybrid_clock.cc:648] HybridClock initialized: now 1790598589296575 us; error 32 us; skew 500 ppm
I20260928 12:29:49.297828 24956 webserver.cc:460] Webserver started at http://127.7.19.66:36887/ using document root <none> and password file <none>
I20260928 12:29:49.298071 24956 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:49.298141 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:49.298293 24956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:49.299321 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "8a79ae9b12a4471fa7a226487dc26fa7"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "a3a4b72c3e2357bc1e57d0785089ffed"
server_key_iv: "4dfdb4bc74251384e4a1f705ebf57d76"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.299674 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "8a79ae9b12a4471fa7a226487dc26fa7"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "a3a4b72c3e2357bc1e57d0785089ffed"
server_key_iv: "4dfdb4bc74251384e4a1f705ebf57d76"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.301015 24956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:29:49.301779 24970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.301950 24956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.302050 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "8a79ae9b12a4471fa7a226487dc26fa7"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "a3a4b72c3e2357bc1e57d0785089ffed"
server_key_iv: "4dfdb4bc74251384e4a1f705ebf57d76"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.302129 24956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:49.334193 24956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:49.334522 24956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:49.334676 24956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:49.334921 24956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:49.335251 24956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:49.335309 24956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.335352 24956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:49.335381 24956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.341147 24956 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:46135
I20260928 12:29:49.341205 25083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:46135 every 8 connection(s)
I20260928 12:29:49.341548 24956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:29:49.346434 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.342324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:36871 (local address 127.7.19.126:39185)
0928 12:29:49.342394 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.342397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.342565 (+ 168us) server_negotiation.cc:407] Connection header received
0928 12:29:49.343045 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.343046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.343057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.343113 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:49.343770 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.344261 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.345247 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.345433 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.345487 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.346208 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:49.346209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:49.346209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:49.346213 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:49.346226 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:49.346230 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:49.346272 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.346355 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.346377 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 12:29:49.346603 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.342230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39185 (local address 127.7.19.66:36871)
0928 12:29:49.342444 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.342477 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.342946 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.343187 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.343190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.343199 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:49.343689 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.343693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.344336 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.344337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.345172 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.345173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.345508 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.345509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.345612 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.346083 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:49.346084 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:49.346111 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:49.346132 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:49.346284 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:49.346288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:49.346290 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.346341 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.346414 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 12:29:49.347045 25084 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39185
I20260928 12:29:49.347142 25084 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:49.347321 25084 heartbeater.cc:511] Master 127.7.19.126:39185 requested a full tablet report, sending...
I20260928 12:29:49.347692 24765 ts_manager.cc:194] Registered new tserver with Master: 8a79ae9b12a4471fa7a226487dc26fa7 (127.7.19.66:46135)
I20260928 12:29:49.348054 24765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.66:36871
I20260928 12:29:49.351321 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 24956
I20260928 12:29:49.351405 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:29:49.351565 7245 external_mini_cluster.cc:1428] Setting key 898e9d0614097d96347dfa527aa3d5c7
I20260928 12:29:49.352528 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39185
--builtin_ntp_servers=127.7.19.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:49.433285 25087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:49.433490 25087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:49.433512 25087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:49.434671 25087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:49.434711 25087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:49.434762 25087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:49.436086 25087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:49.436329 25087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:49.436621 25087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:49.441488 25095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.441498 25093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:49.441498 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:49.441783 25087 server_base.cc:1034] running on GCE node
I20260928 12:29:49.442036 25087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:49.442225 25087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:49.443403 25087 hybrid_clock.cc:648] HybridClock initialized: now 1790598589443388 us; error 47 us; skew 500 ppm
I20260928 12:29:49.445070 25087 webserver.cc:460] Webserver started at http://127.7.19.67:36481/ using document root <none> and password file <none>
I20260928 12:29:49.445338 25087 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:49.445400 25087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:49.445525 25087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:49.446769 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "b4ca37dc517b252abc947f3e606ed51e"
server_key_iv: "0b3f62d63d9e1123e9624c37737981f5"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.447077 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "b4ca37dc517b252abc947f3e606ed51e"
server_key_iv: "0b3f62d63d9e1123e9624c37737981f5"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.448285 25087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:49.449035 25101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.449234 25087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.449307 25087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c"
format_stamp: "Formatted at 2026-09-28 12:29:49 on dist-test-slave-cb7x"
server_key: "b4ca37dc517b252abc947f3e606ed51e"
server_key_iv: "0b3f62d63d9e1123e9624c37737981f5"
server_key_version: "encryptionkey@0"
I20260928 12:29:49.449395 25087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:49.461499 25087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:49.461771 25087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:49.461912 25087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:49.462122 25087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:49.462437 25087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:49.462491 25087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.462531 25087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:49.462556 25087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:49.468254 25087 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:32783
I20260928 12:29:49.468303 25214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:32783 every 8 connection(s)
I20260928 12:29:49.468617 25087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:29:49.473846 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.469382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:41087 (local address 127.7.19.126:39185)
0928 12:29:49.469425 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.469427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.469702 (+ 275us) server_negotiation.cc:407] Connection header received
0928 12:29:49.470232 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.470234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.470246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.470298 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:49.470973 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.471474 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.472484 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.472697 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.472776 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.473524 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:49.473525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:49.473526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:49.473530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:49.473544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:49.473548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:49.473597 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.473736 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.473769 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:29:49.474077 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.469387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:39185 (local address 127.7.19.67:41087)
0928 12:29:49.469583 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.469615 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.470129 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.470375 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.470379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.470388 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:49.470868 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.470873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.471616 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.471617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.472376 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.472378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.472782 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.472784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.472912 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.473381 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:49.473381 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:49.473408 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:49.473429 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:49.473655 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:49.473659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:49.473662 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.473737 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.473833 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260928 12:29:49.474565 25215 heartbeater.cc:347] Connected to a master server at 127.7.19.126:39185
I20260928 12:29:49.474634 25215 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:49.474810 25215 heartbeater.cc:511] Master 127.7.19.126:39185 requested a full tablet report, sending...
I20260928 12:29:49.475183 24765 ts_manager.cc:194] Registered new tserver with Master: 232ff3ed9fc24f1f8615cba8ee6bd00c (127.7.19.67:32783)
I20260928 12:29:49.475559 24765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.67:41087
I20260928 12:29:49.477101 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25087
I20260928 12:29:49.477195 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:29:49.477355 7245 external_mini_cluster.cc:1428] Setting key 9ee01df67b510f0096be55144a44ff34
I20260928 12:29:49.478219 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:49.482487 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.479433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.7.19.126:39185)
0928 12:29:49.479490 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.479493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.479613 (+ 120us) server_negotiation.cc:407] Connection header received
0928 12:29:49.479687 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.479688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.479706 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.479775 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:49.479953 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.480397 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.481342 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.481528 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.481610 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.482252 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:49.482254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:49.482254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:49.482259 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:49.482271 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:49.482275 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:49.482328 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.482387 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.482408 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:29:49.483925 24765 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:49.485505 24765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:49.491523 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.487494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:46135 (local address 127.0.0.1:44386)
0928 12:29:49.487664 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.487704 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.487767 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.487959 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.487962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.487965 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:49.488025 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.488026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.489422 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.489423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.490375 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.490377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.491229 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.491305 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.491319 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.491399 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.491447 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":64,"thread_start_us":69,"threads_started":1}
I20260928 12:29:49.491531 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.487580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.7.19.66:46135)
0928 12:29:49.487798 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.487803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.487815 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:29:49.487854 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.487856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.487870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.487953 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:49.488118 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.489321 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.490469 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.491140 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491221 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.491309 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.491411 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.491450 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260928 12:29:49.491971 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.487721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:42555 (local address 127.0.0.1:32820)
0928 12:29:49.488045 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.488056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.488098 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.488354 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.488355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.488357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:49.488394 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.488395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.489653 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.489653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.490755 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.490758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.491738 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.491794 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.491798 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.491843 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.491886 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260928 12:29:49.492100 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.487508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:32783 (local address 127.0.0.1:32980)
0928 12:29:49.487859 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.487869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.487919 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.488382 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.488383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.488386 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:49.488434 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.488436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.489416 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.489417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.490789 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.490792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.491859 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.491922 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.491927 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.491989 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.492029 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":115,"thread_start_us":95,"threads_started":1}
I20260928 12:29:49.492364 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.487931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.7.19.65:42555)
0928 12:29:49.488094 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.488099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.488109 (+ 10us) server_negotiation.cc:407] Connection header received
0928 12:29:49.488165 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.488167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.488180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.488283 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:49.488469 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.489574 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.490854 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.491535 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491597 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.491706 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.492221 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.492268 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260928 12:29:49.492112 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.487591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.7.19.67:32783)
0928 12:29:49.488203 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.488207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.488214 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:29:49.488265 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.488267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.488281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.488508 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:49.488561 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.489330 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491035 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.491757 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.491856 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.491937 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.492002 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.492036 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":100,"threads_started":1}
I20260928 12:29:49.492560 25018 tablet_service.cc:1467] Processing CreateTablet for tablet c49e5a49e8d04704b7057c759e97f8b5 (DEFAULT_TABLE table=test-table [id=f3afeec318ab4937bcfc596a4b2c3fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:49.492970 25018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49e5a49e8d04704b7057c759e97f8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:49.493106 24887 tablet_service.cc:1467] Processing CreateTablet for tablet c49e5a49e8d04704b7057c759e97f8b5 (DEFAULT_TABLE table=test-table [id=f3afeec318ab4937bcfc596a4b2c3fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:49.493309 25149 tablet_service.cc:1467] Processing CreateTablet for tablet c49e5a49e8d04704b7057c759e97f8b5 (DEFAULT_TABLE table=test-table [id=f3afeec318ab4937bcfc596a4b2c3fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:49.493441 24887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49e5a49e8d04704b7057c759e97f8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:49.493628 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c49e5a49e8d04704b7057c759e97f8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:49.495572 25229 tablet_bootstrap.cc:492] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: Bootstrap starting.
I20260928 12:29:49.496181 25229 tablet_bootstrap.cc:654] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:49.496512 25229 log.cc:824] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:49.496629 25230 tablet_bootstrap.cc:492] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Bootstrap starting.
I20260928 12:29:49.497323 25230 tablet_bootstrap.cc:654] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:49.497407 25229 tablet_bootstrap.cc:492] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: No bootstrap required, opened a new log
I20260928 12:29:49.497483 25229 ts_tablet_manager.cc:1397] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:49.497664 25230 log.cc:824] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:49.497998 25231 tablet_bootstrap.cc:492] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578: Bootstrap starting.
I20260928 12:29:49.498318 25230 tablet_bootstrap.cc:492] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: No bootstrap required, opened a new log
I20260928 12:29:49.498396 25230 ts_tablet_manager.cc:1397] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:49.498733 25231 tablet_bootstrap.cc:654] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:49.499032 25231 log.cc:824] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:49.499780 25231 tablet_bootstrap.cc:492] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578: No bootstrap required, opened a new log
I20260928 12:29:49.499852 25231 ts_tablet_manager.cc:1397] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:49.500192 25229 raft_consensus.cc:348] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.500399 25229 raft_consensus.cc:374] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:49.500437 25229 raft_consensus.cc:729] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a79ae9b12a4471fa7a226487dc26fa7, State: Initialized, Role: FOLLOWER
I20260928 12:29:49.500631 25229 consensus_queue.cc:260] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.500748 25230 raft_consensus.cc:348] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.500916 25230 raft_consensus.cc:374] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:49.500921 25229 ts_tablet_manager.cc:1428] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 12:29:49.500959 25230 raft_consensus.cc:729] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232ff3ed9fc24f1f8615cba8ee6bd00c, State: Initialized, Role: FOLLOWER
I20260928 12:29:49.501044 25084 heartbeater.cc:503] Master 127.7.19.126:39185 was elected leader, sending a full tablet report...
I20260928 12:29:49.501379 25230 consensus_queue.cc:260] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.501665 25230 ts_tablet_manager.cc:1428] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:29:49.501713 25215 heartbeater.cc:503] Master 127.7.19.126:39185 was elected leader, sending a full tablet report...
I20260928 12:29:49.502458 25231 raft_consensus.cc:348] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.502630 25231 raft_consensus.cc:374] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:49.502712 25231 raft_consensus.cc:729] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f828485f1bf34269b0b7e550be166578, State: Initialized, Role: FOLLOWER
I20260928 12:29:49.502851 25231 consensus_queue.cc:260] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.503111 25231 ts_tablet_manager.cc:1428] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 12:29:49.503149 24953 heartbeater.cc:503] Master 127.7.19.126:39185 was elected leader, sending a full tablet report...
I20260928 12:29:49.529971 25235 raft_consensus.cc:482] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:49.530093 25235 raft_consensus.cc:504] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.530395 25235 leader_election.cc:288] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 232ff3ed9fc24f1f8615cba8ee6bd00c (127.7.19.67:32783), f828485f1bf34269b0b7e550be166578 (127.7.19.65:42555)
I20260928 12:29:49.533409 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.530780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:48909 (local address 127.7.19.67:32783)
0928 12:29:49.530849 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.530852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.530863 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:29:49.530896 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.530897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.530921 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.531014 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:49.531230 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.531704 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.532704 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.533058 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.533144 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.533155 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.533301 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.533341 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 12:29:49.533409 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.530660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:32783 (local address 127.7.19.66:48909)
0928 12:29:49.530747 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.530761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.530803 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.531062 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.531064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.531066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:49.531140 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.531142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.531788 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.531789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.532584 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.532586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.533142 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.533143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.533214 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.533217 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.533295 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.533335 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 12:29:49.533700 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.530929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:48715 (local address 127.7.19.65:42555)
0928 12:29:49.530998 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.531000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.531190 (+ 190us) server_negotiation.cc:407] Connection header received
0928 12:29:49.531251 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.531252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.531266 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.531317 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:49.531478 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.531932 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.532925 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.533297 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.533379 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.533388 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.533571 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.533622 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:29:49.533747 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49e5a49e8d04704b7057c759e97f8b5" candidate_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" is_pre_election: true
I20260928 12:29:49.533922 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.530854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:42555 (local address 127.7.19.66:48715)
0928 12:29:49.531087 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.531100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:49.531195 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:49.531326 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:49.531327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:49.531328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:49.531403 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.531404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.532013 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.532013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.532823 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:49.532825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.533380 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.533381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:49.533450 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.533455 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 12:29:49.533800 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 12:29:49.533848 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":175,"threads_started":1}
I20260928 12:29:49.533968 25169 raft_consensus.cc:2454] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a79ae9b12a4471fa7a226487dc26fa7 in term 0.
I20260928 12:29:49.534198 24974 leader_election.cc:302] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232ff3ed9fc24f1f8615cba8ee6bd00c, 8a79ae9b12a4471fa7a226487dc26fa7; no voters:
I20260928 12:29:49.534236 24907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49e5a49e8d04704b7057c759e97f8b5" candidate_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f828485f1bf34269b0b7e550be166578" is_pre_election: true
I20260928 12:29:49.534371 24907 raft_consensus.cc:2454] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a79ae9b12a4471fa7a226487dc26fa7 in term 0.
I20260928 12:29:49.534350 25235 raft_consensus.cc:2781] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:49.534421 25235 raft_consensus.cc:482] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:49.534458 25235 raft_consensus.cc:3037] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:49.535164 25235 raft_consensus.cc:504] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.535297 25235 leader_election.cc:288] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [CANDIDATE]: Term 1 election: Requested vote from peers 232ff3ed9fc24f1f8615cba8ee6bd00c (127.7.19.67:32783), f828485f1bf34269b0b7e550be166578 (127.7.19.65:42555)
I20260928 12:29:49.535487 24907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49e5a49e8d04704b7057c759e97f8b5" candidate_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f828485f1bf34269b0b7e550be166578"
I20260928 12:29:49.535497 25169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c49e5a49e8d04704b7057c759e97f8b5" candidate_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c"
I20260928 12:29:49.535583 25169 raft_consensus.cc:3037] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:49.535559 24907 raft_consensus.cc:3037] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:49.536263 25169 raft_consensus.cc:2454] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a79ae9b12a4471fa7a226487dc26fa7 in term 1.
I20260928 12:29:49.536374 24907 raft_consensus.cc:2454] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a79ae9b12a4471fa7a226487dc26fa7 in term 1.
I20260928 12:29:49.536473 24974 leader_election.cc:302] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232ff3ed9fc24f1f8615cba8ee6bd00c, 8a79ae9b12a4471fa7a226487dc26fa7; no voters:
I20260928 12:29:49.536559 25235 raft_consensus.cc:2781] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:49.536767 25235 raft_consensus.cc:686] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 1 LEADER]: Becoming Leader. State: Replica: 8a79ae9b12a4471fa7a226487dc26fa7, State: Running, Role: LEADER
I20260928 12:29:49.536886 25235 consensus_queue.cc:237] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } }
I20260928 12:29:49.537746 24765 catalog_manager.cc:5495] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a79ae9b12a4471fa7a226487dc26fa7 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a79ae9b12a4471fa7a226487dc26fa7" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 46135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 } health_report { overall_health: UNKNOWN } } }
I20260928 12:29:49.553355 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.550884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.7.19.66:46135)
0928 12:29:49.550925 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:49.550927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:49.550936 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:49.551019 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:49.551020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:49.551037 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:49.551096 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:49.551242 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.551681 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.552550 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:49.552828 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:49.552883 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:49.553061 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:49.553150 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:49.553208 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:49.553222 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 12:29:49.553240 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:29:49.555094 25169 raft_consensus.cc:1261] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 1 FOLLOWER]: Refusing update from remote peer 8a79ae9b12a4471fa7a226487dc26fa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:49.555115 24907 raft_consensus.cc:1261] T c49e5a49e8d04704b7057c759e97f8b5 P f828485f1bf34269b0b7e550be166578 [term 1 FOLLOWER]: Refusing update from remote peer 8a79ae9b12a4471fa7a226487dc26fa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:49.555460 25239 consensus_queue.cc:1035] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "232ff3ed9fc24f1f8615cba8ee6bd00c" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:49.555599 25235 consensus_queue.cc:1035] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f828485f1bf34269b0b7e550be166578" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:49.558097 25243 mvcc.cc:205] Tried to move back new op lower bound from 7334291822814994432 to 7334291822743707648. Current Snapshot: MvccSnapshot[applied={T|T < 7334291822814994432}]
I20260928 12:29:49.558172 25244 mvcc.cc:205] Tried to move back new op lower bound from 7334291822814994432 to 7334291822743707648. Current Snapshot: MvccSnapshot[applied={T|T < 7334291822814994432}]
I20260928 12:29:49.560050 24764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:49.560158 24764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:49.560933 24764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19e552c4332342378ab4cee295d3dff1: Sending DeleteTablet for 3 replicas of tablet c49e5a49e8d04704b7057c759e97f8b5
I20260928 12:29:49.561309 25018 tablet_service.cc:1514] Processing DeleteTablet for tablet c49e5a49e8d04704b7057c759e97f8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:49 UTC) from {username='slave'} at 127.0.0.1:44386
I20260928 12:29:49.561506 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24825
I20260928 12:29:49.561571 24887 tablet_service.cc:1514] Processing DeleteTablet for tablet c49e5a49e8d04704b7057c759e97f8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:49 UTC) from {username='slave'} at 127.0.0.1:32820
I20260928 12:29:49.562050 25149 tablet_service.cc:1514] Processing DeleteTablet for tablet c49e5a49e8d04704b7057c759e97f8b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:49 UTC) from {username='slave'} at 127.0.0.1:32980
I20260928 12:29:49.562256 25253 tablet_replica.cc:330] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: stopping tablet replica
I20260928 12:29:49.562386 25253 raft_consensus.cc:2229] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:49.562521 25253 raft_consensus.cc:2258] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:49.563023 25253 ts_tablet_manager.cc:1905] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:49.564877 25253 ts_tablet_manager.cc:1918] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:49.564989 25253 log.cc:1196] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/c49e5a49e8d04704b7057c759e97f8b5
I20260928 12:29:49.565279 25253 ts_tablet_manager.cc:1939] T c49e5a49e8d04704b7057c759e97f8b5 P 232ff3ed9fc24f1f8615cba8ee6bd00c: Deleting consensus metadata
I20260928 12:29:49.565712 24752 catalog_manager.cc:4841] TS 232ff3ed9fc24f1f8615cba8ee6bd00c (127.7.19.67:32783): tablet c49e5a49e8d04704b7057c759e97f8b5 (table test-table [id=f3afeec318ab4937bcfc596a4b2c3fe3]) successfully deleted
W20260928 12:29:49.567512 24752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:42555 (error 108)
I20260928 12:29:49.568082 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24956
I20260928 12:29:49.568113 25252 tablet_replica.cc:330] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7: stopping tablet replica
I20260928 12:29:49.568229 25252 raft_consensus.cc:2229] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:29:49.568356 25252 raft_consensus.cc:2258] T c49e5a49e8d04704b7057c759e97f8b5 P 8a79ae9b12a4471fa7a226487dc26fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:49.573079 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.572948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:42555 (local address 127.0.0.1:32836)
0928 12:29:49.572996 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.573045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 12:29:49.573170 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:49.573068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:46135 (local address 127.0.0.1:44414)
0928 12:29:49.573101 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:49.573124 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.66:46135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 12:29:49.573215 24752 catalog_manager.cc:4584] TS f828485f1bf34269b0b7e550be166578 (127.7.19.65:42555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c49e5a49e8d04704b7057c759e97f8b5: Network error: Client connection negotiation failed: client connection to 127.7.19.65:42555: connect: Connection refused (error 111)
I20260928 12:29:49.573362 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25087
I20260928 12:29:49.577935 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 24734
2026-09-28T12:29:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (680 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:49 dist-test-slave-cb7x krb5kdc[25257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:49 dist-test-slave-cb7x krb5kdc[25257](info): set up 2 sockets
Sep 28 12:29:49 dist-test-slave-cb7x krb5kdc[25257](info): commencing operation
krb5kdc: starting...
W20260928 12:29:50.612434 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:29:50 dist-test-slave-cb7x krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:29:50.735606 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43503
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:50.815811 25270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:50.816056 25270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:50.816090 25270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:50.817265 25270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:50.817338 25270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:50.817363 25270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:50.817386 25270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:50.817414 25270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:50.818892 25270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:50.819171 25270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:50.819397 25270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:50.824208 25276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:50.824262 25275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:50.824442 25270 server_base.cc:1034] running on GCE node
W20260928 12:29:50.824216 25278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:50.824784 25270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:50.825022 25270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:50.826159 25270 hybrid_clock.cc:648] HybridClock initialized: now 1790598590826148 us; error 28 us; skew 500 ppm
Sep 28 12:29:50 dist-test-slave-cb7x krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598590, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:50.828648 25270 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:29:50.829185 25270 webserver.cc:460] Webserver started at http://127.7.19.126:46323/ using document root <none> and password file <none>
I20260928 12:29:50.829414 25270 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:50.829460 25270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:50.829591 25270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:50.830646 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "066d23e571cb4cb5b615e7f420f68d5a"
format_stamp: "Formatted at 2026-09-28 12:29:50 on dist-test-slave-cb7x"
server_key: "d472d9f946039f8c39754c480faebf5e"
server_key_iv: "94ddf96128914f21b03f4e6579f5ac87"
server_key_version: "encryptionkey@0"
I20260928 12:29:50.831003 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "066d23e571cb4cb5b615e7f420f68d5a"
format_stamp: "Formatted at 2026-09-28 12:29:50 on dist-test-slave-cb7x"
server_key: "d472d9f946039f8c39754c480faebf5e"
server_key_iv: "94ddf96128914f21b03f4e6579f5ac87"
server_key_version: "encryptionkey@0"
I20260928 12:29:50.832337 25270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:29:50.833101 25285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:50.833288 25270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:50.833379 25270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "066d23e571cb4cb5b615e7f420f68d5a"
format_stamp: "Formatted at 2026-09-28 12:29:50 on dist-test-slave-cb7x"
server_key: "d472d9f946039f8c39754c480faebf5e"
server_key_iv: "94ddf96128914f21b03f4e6579f5ac87"
server_key_version: "encryptionkey@0"
I20260928 12:29:50.833447 25270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:50.851505 25270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:50.853667 25270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:50.853832 25270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:50.858111 25270 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:43503
I20260928 12:29:50.858150 25337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:43503 every 8 connection(s)
I20260928 12:29:50.858525 25270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:50.859390 25338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:50.860270 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25270
I20260928 12:29:50.860395 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:50.860590 7245 external_mini_cluster.cc:1428] Setting key fe58f3d36c29b5a6135f666225849574
I20260928 12:29:50.862730 25338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a: Bootstrap starting.
I20260928 12:29:50.863328 25338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:50.863632 25338 log.cc:824] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:50.864429 25338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a: No bootstrap required, opened a new log
I20260928 12:29:50.866982 25338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066d23e571cb4cb5b615e7f420f68d5a" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43503 } }
I20260928 12:29:50.867164 25338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:50.867192 25338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066d23e571cb4cb5b615e7f420f68d5a, State: Initialized, Role: FOLLOWER
I20260928 12:29:50.867328 25338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066d23e571cb4cb5b615e7f420f68d5a" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43503 } }
I20260928 12:29:50.867408 25338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:50.867461 25338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:50.867509 25338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 0 FOLLOWER]: Advancing to term 1
Sep 28 12:29:50 dist-test-slave-cb7x krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:50.868134 25338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066d23e571cb4cb5b615e7f420f68d5a" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43503 } }
I20260928 12:29:50.868266 25338 leader_election.cc:302] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 066d23e571cb4cb5b615e7f420f68d5a; no voters:
I20260928 12:29:50.868438 25338 leader_election.cc:288] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:50.868503 25343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:50.868714 25343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [term 1 LEADER]: Becoming Leader. State: Replica: 066d23e571cb4cb5b615e7f420f68d5a, State: Running, Role: LEADER
I20260928 12:29:50.868733 25338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:50.868805 25343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066d23e571cb4cb5b615e7f420f68d5a" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43503 } }
I20260928 12:29:50.869228 25344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "066d23e571cb4cb5b615e7f420f68d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066d23e571cb4cb5b615e7f420f68d5a" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43503 } } }
I20260928 12:29:50.869349 25344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:50.869524 25345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 066d23e571cb4cb5b615e7f420f68d5a. Latest consensus state: current_term: 1 leader_uuid: "066d23e571cb4cb5b615e7f420f68d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "066d23e571cb4cb5b615e7f420f68d5a" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 43503 } } }
I20260928 12:29:50.869661 25345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:50.870082 25358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:50.870779 25358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:50.872110 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:50.861133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.7.19.126:43503)
0928 12:29:50.861305 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:50.861313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:50.861339 (+ 26us) server_negotiation.cc:407] Connection header received
0928 12:29:50.861748 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:50.861754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:50.861799 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:50.861913 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:50.862739 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:50.863806 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:50.864848 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:50.865264 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:50.865345 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:50.868219 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:50.868224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:50.868224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:50.868237 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:50.870415 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:50.870825 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:50.870828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:50.870828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:50.870930 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:50.871153 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:50.871154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:50.871154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:50.871457 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:50.871617 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:50.871747 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 12:29:50.871888 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 12:29:50.872792 25358 catalog_manager.cc:1347] Generated new cluster ID: 8e9254c6c64c44abbcb5bfa14bd088fc
I20260928 12:29:50.872848 25358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:50.894411 25358 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:50.895166 25358 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:50.911346 25358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a: Generated new TSK 0
I20260928 12:29:50.911576 25358 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:29:50.939589 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43503
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:51.020074 25364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:51.020288 25364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:51.020311 25364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:51.021521 25364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:51.021569 25364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:51.021597 25364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:51.023068 25364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:51.023326 25364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:51.023571 25364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:51.028724 25372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:51.028724 25369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:51.028752 25364 server_base.cc:1034] running on GCE node
W20260928 12:29:51.028741 25370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:51.029119 25364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:51.029368 25364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:51.030512 25364 hybrid_clock.cc:648] HybridClock initialized: now 1790598591030490 us; error 40 us; skew 500 ppm
Sep 28 12:29:51 dist-test-slave-cb7x krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598591, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:51.033102 25364 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:29:51.033552 25364 webserver.cc:460] Webserver started at http://127.7.19.65:46571/ using document root <none> and password file <none>
I20260928 12:29:51.033804 25364 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:51.033874 25364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:51.034022 25364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:51.035086 25364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "c9181479154f4f85a480a13e1a8f5d52"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "47a9ebdfa26ee7f5b21b8f59be5d98d0"
server_key_iv: "fc7622db54786b9879cb24ea2ac4819d"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.035455 25364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "c9181479154f4f85a480a13e1a8f5d52"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "47a9ebdfa26ee7f5b21b8f59be5d98d0"
server_key_iv: "fc7622db54786b9879cb24ea2ac4819d"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.037001 25364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:29:51.037770 25379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.037945 25364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:51.038046 25364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "c9181479154f4f85a480a13e1a8f5d52"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "47a9ebdfa26ee7f5b21b8f59be5d98d0"
server_key_iv: "fc7622db54786b9879cb24ea2ac4819d"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.038125 25364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:51.046806 25364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:51.049228 25364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:51.049392 25364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:51.049615 25364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:51.049924 25364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:51.049993 25364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.050030 25364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:51.050058 25364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.056108 25364 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:33487
I20260928 12:29:51.056191 25492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:33487 every 8 connection(s)
I20260928 12:29:51.056484 25364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
Sep 28 12:29:51 dist-test-slave-cb7x krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598591, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:51.064433 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25364
I20260928 12:29:51.064543 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:29:51.064802 7245 external_mini_cluster.cc:1428] Setting key 6d83c1f58844cddf9831a5739477b2fa
I20260928 12:29:51.066043 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.057230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:47839 (local address 127.7.19.126:43503)
0928 12:29:51.057303 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.057306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.057516 (+ 210us) server_negotiation.cc:407] Connection header received
0928 12:29:51.058043 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.058045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.058056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.058112 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:51.059022 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.059525 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.060514 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.060835 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.060903 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.063062 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:51.063063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:51.063063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:51.063071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:51.064397 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.064900 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.064902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.064902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.064990 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.065327 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.065328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.065328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.065467 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:51.065544 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.065890 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.065977 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:29:51.066354 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.057215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43503 (local address 127.7.19.65:47839)
0928 12:29:51.057382 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.057423 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.057943 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.058174 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.058178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.058742 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:51.058937 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.058941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.059603 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.059603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.060411 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.060414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.060912 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.060912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.061017 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.061495 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:51.061502 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:51.062969 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:51.064480 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:51.064486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:51.064487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:51.064819 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:51.065070 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:51.065071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:51.065072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:51.065255 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:51.065558 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:51.065565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:51.065742 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.065865 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.066045 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260928 12:29:51.066891 25493 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43503
I20260928 12:29:51.067026 25493 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:51.067215 25493 heartbeater.cc:511] Master 127.7.19.126:43503 requested a full tablet report, sending...
I20260928 12:29:51.067847 25302 ts_manager.cc:194] Registered new tserver with Master: c9181479154f4f85a480a13e1a8f5d52 (127.7.19.65:33487)
I20260928 12:29:51.068614 25302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:47839
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:29:51.091747 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43503
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:51.173861 25498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:51.174095 25498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:51.174130 25498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:51.175235 25498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:51.175300 25498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:51.175348 25498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:51.176791 25498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:51.177047 25498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:51.177290 25498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:51.182430 25504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:51.182472 25503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:51.182435 25506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:51.182781 25498 server_base.cc:1034] running on GCE node
I20260928 12:29:51.182999 25498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:51.183234 25498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:51.184399 25498 hybrid_clock.cc:648] HybridClock initialized: now 1790598591184391 us; error 40 us; skew 500 ppm
Sep 28 12:29:51 dist-test-slave-cb7x krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598591, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:51.187325 25498 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:29:51.187826 25498 webserver.cc:460] Webserver started at http://127.7.19.66:39939/ using document root <none> and password file <none>
I20260928 12:29:51.188086 25498 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:51.188164 25498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:51.188300 25498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:51.189424 25498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "6dc0e6d29ebe45e993558ad4ba0264ad"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "719ea2ea7d76bb1176568361479282a0"
server_key_iv: "23abfaaea8da72b5b32d3244e8ea6945"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.189935 25498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "6dc0e6d29ebe45e993558ad4ba0264ad"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "719ea2ea7d76bb1176568361479282a0"
server_key_iv: "23abfaaea8da72b5b32d3244e8ea6945"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.191375 25498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:29:51.192137 25513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.192325 25498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:51.192431 25498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "6dc0e6d29ebe45e993558ad4ba0264ad"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "719ea2ea7d76bb1176568361479282a0"
server_key_iv: "23abfaaea8da72b5b32d3244e8ea6945"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.192505 25498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:51.204517 25498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:51.206943 25498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:51.207113 25498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:51.207346 25498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:51.207660 25498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:51.207715 25498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.207754 25498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:51.207782 25498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.213654 25498 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:34095
I20260928 12:29:51.213703 25626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:34095 every 8 connection(s)
I20260928 12:29:51.214015 25498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:29:51.216427 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25498
I20260928 12:29:51.216532 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:29:51.216749 7245 external_mini_cluster.cc:1428] Setting key 5bb488c0575c913b5c7ca94b6db8a88a
Sep 28 12:29:51 dist-test-slave-cb7x krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598591, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:51.225050 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.214792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:34157 (local address 127.7.19.126:43503)
0928 12:29:51.214904 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.214907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.215085 (+ 178us) server_negotiation.cc:407] Connection header received
0928 12:29:51.215629 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.215631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.215644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.215686 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:51.216942 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.217679 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.219236 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.219448 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.219527 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.221903 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:51.221905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:51.221906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:51.221914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:51.223593 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.224004 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.224005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.224006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.224069 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.224306 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.224307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.224308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.224459 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:51.224534 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.224838 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.224920 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 12:29:51.225243 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.214691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43503 (local address 127.7.19.66:34157)
0928 12:29:51.214947 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.214997 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.215515 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.215776 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.215781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.216471 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:51.216815 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.216822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.217775 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.217776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.219133 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.219136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.219532 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.219533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.219678 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.220172 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:51.220183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:51.221803 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:51.223686 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:51.223692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:51.223694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:51.223926 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:51.224144 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:51.224144 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:51.224145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:51.224240 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:51.224525 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:51.224530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:51.224644 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.224801 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.224988 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260928 12:29:51.225705 25627 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43503
I20260928 12:29:51.225832 25627 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
I20260928 12:29:51.226020 25627 heartbeater.cc:511] Master 127.7.19.126:43503 requested a full tablet report, sending...
I20260928 12:29:51.226418 25302 ts_manager.cc:194] Registered new tserver with Master: 6dc0e6d29ebe45e993558ad4ba0264ad (127.7.19.66:34095)
I20260928 12:29:51.226806 25302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:34157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:29:51.244364 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43503
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:51.325382 25632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:51.325574 25632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:51.325596 25632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:51.326669 25632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:51.326704 25632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:51.326730 25632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:51.328102 25632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:51.328286 25632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:51.328461 25632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:51.333650 25637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:51.333662 25638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:51.333909 25640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:51.333917 25632 server_base.cc:1034] running on GCE node
I20260928 12:29:51.334192 25632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:51.334443 25632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:51.335608 25632 hybrid_clock.cc:648] HybridClock initialized: now 1790598591335651 us; error 89 us; skew 500 ppm
Sep 28 12:29:51 dist-test-slave-cb7x krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598591, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:51.338335 25632 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:29:51.338716 25632 webserver.cc:460] Webserver started at http://127.7.19.67:45453/ using document root <none> and password file <none>
I20260928 12:29:51.338917 25632 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:51.338959 25632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:51.339049 25632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:51.340037 25632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "ff10665d9df04ed48ef610c619b1f07f"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "8986ebbe8e6cc6387501bb1428b7744e"
server_key_iv: "514873fb21e63b558c912c863f39153f"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.340338 25632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "ff10665d9df04ed48ef610c619b1f07f"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "8986ebbe8e6cc6387501bb1428b7744e"
server_key_iv: "514873fb21e63b558c912c863f39153f"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.341809 25632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:29:51.342546 25647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.342826 25632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:51.342901 25632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "ff10665d9df04ed48ef610c619b1f07f"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "8986ebbe8e6cc6387501bb1428b7744e"
server_key_iv: "514873fb21e63b558c912c863f39153f"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.342998 25632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:51.353750 25632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:51.356135 25632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:51.356333 25632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:51.356608 25632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:51.356963 25632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:51.357020 25632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.357090 25632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:51.357131 25632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.362775 25632 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:36273
I20260928 12:29:51.362864 25760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:36273 every 8 connection(s)
I20260928 12:29:51.363179 25632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:29:51 dist-test-slave-cb7x krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598591, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:51.369904 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25632
I20260928 12:29:51.370028 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:29:51.370265 7245 external_mini_cluster.cc:1428] Setting key a3acc194a446ec125f2b913e029d5e64
I20260928 12:29:51.373005 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.363911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:58365 (local address 127.7.19.126:43503)
0928 12:29:51.363960 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.363962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.364159 (+ 197us) server_negotiation.cc:407] Connection header received
0928 12:29:51.364725 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.364727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.364740 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.364791 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:51.365785 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.366293 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.367321 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.367529 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.367618 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.370196 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:51.370197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:51.370197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:51.370206 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:51.371743 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.372123 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.372124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.372125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.372210 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.372434 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.372435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.372435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.372540 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:51.372590 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.372883 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.372945 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:29:51.373183 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.363867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:43503 (local address 127.7.19.67:58365)
0928 12:29:51.364010 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.364056 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.364586 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.364881 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.364887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.365463 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:51.365651 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.365656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.366401 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.366402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.367224 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.367226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.367627 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.367628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.367768 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.368367 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:51.368376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:51.370078 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:51.371815 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:51.371821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:51.371823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:51.372047 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:51.372280 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:51.372280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:51.372280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:51.372377 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:51.372581 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:51.372583 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:51.372770 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.372856 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.372973 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 12:29:51.373683 25761 heartbeater.cc:347] Connected to a master server at 127.7.19.126:43503
I20260928 12:29:51.373790 25761 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:51.373968 25761 heartbeater.cc:511] Master 127.7.19.126:43503 requested a full tablet report, sending...
I20260928 12:29:51.374294 25302 ts_manager.cc:194] Registered new tserver with Master: ff10665d9df04ed48ef610c619b1f07f (127.7.19.67:36273)
I20260928 12:29:51.374665 25302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:58365
I20260928 12:29:51.382134 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:51.389816 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.383548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.7.19.126:43503)
0928 12:29:51.383604 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.383606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.383705 (+ 99us) server_negotiation.cc:407] Connection header received
0928 12:29:51.383742 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.383743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.383752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.383805 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:51.384473 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.385060 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.385896 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.386074 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.386128 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.387370 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:51.387371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:51.387371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:51.387378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:51.388637 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.389122 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.389123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.389124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.389189 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:51.389381 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:51.389382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:51.389382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:51.389499 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:51.389576 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.389700 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.389752 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:29:51.391350 25302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:51.392938 25302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:51.399544 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.395205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:34095 (local address 127.0.0.1:39166)
0928 12:29:51.395538 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.395593 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.395670 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.396009 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.396011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.396014 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:51.396085 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.396087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.397462 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.397463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.398562 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.398565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.399235 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.399236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.399314 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.399327 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.399392 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.399449 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":5,"thread_start_us":57,"threads_started":1}
I20260928 12:29:51.399678 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.395480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.7.19.65:33487)
0928 12:29:51.395713 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.395717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.395889 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:29:51.395939 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.395942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.395957 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.396032 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:51.396264 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.397470 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.398612 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.399318 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.399390 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.399452 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.399528 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.399576 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":166,"threads_started":1}
I20260928 12:29:51.399681 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.395214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:33487 (local address 127.0.0.1:48546)
0928 12:29:51.395751 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.395760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.395823 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.396127 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.396128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.396129 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:51.396202 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.396204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.397563 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.397565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.398500 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.398503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.399399 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.399400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.399455 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.399458 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.399515 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.399543 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":49,"threads_started":1}
I20260928 12:29:51.400022 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.395219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36273 (local address 127.0.0.1:44106)
0928 12:29:51.395896 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.395905 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.395945 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.396225 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.396226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.396228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:51.396264 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.396265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.397220 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.397221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.398565 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.398566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.399581 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.399582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.399837 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.399841 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.399897 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.399934 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":5,"thread_start_us":50,"threads_started":1}
I20260928 12:29:51.400085 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.395459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.7.19.67:36273)
0928 12:29:51.395636 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.395641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.396003 (+ 362us) server_negotiation.cc:407] Connection header received
0928 12:29:51.396048 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.396050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.396064 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.396131 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:51.396321 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.397120 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.398646 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.399320 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.399394 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.399452 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.399946 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.399986 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260928 12:29:51.400657 25695 tablet_service.cc:1467] Processing CreateTablet for tablet b07e636211e7411eaac8d1a9c51bcca5 (DEFAULT_TABLE table=test-table [id=70f2b5a5e0b248d4949f9319e241e47d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:51.400624 25427 tablet_service.cc:1467] Processing CreateTablet for tablet b07e636211e7411eaac8d1a9c51bcca5 (DEFAULT_TABLE table=test-table [id=70f2b5a5e0b248d4949f9319e241e47d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:51.401017 25695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07e636211e7411eaac8d1a9c51bcca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:51.401018 25427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07e636211e7411eaac8d1a9c51bcca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:51.404448 25775 tablet_bootstrap.cc:492] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Bootstrap starting.
I20260928 12:29:51.404556 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.395291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.7.19.66:34095)
0928 12:29:51.395523 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.395528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.395787 (+ 259us) server_negotiation.cc:407] Connection header received
0928 12:29:51.395833 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.395835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.395850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.395922 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:51.396157 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.397365 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.398662 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.399139 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.399224 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.399286 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.400216 (+ 930us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.400263 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":137,"threads_started":1}
I20260928 12:29:51.405328 25775 tablet_bootstrap.cc:654] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:51.405356 25561 tablet_service.cc:1467] Processing CreateTablet for tablet b07e636211e7411eaac8d1a9c51bcca5 (DEFAULT_TABLE table=test-table [id=70f2b5a5e0b248d4949f9319e241e47d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:51.405743 25561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07e636211e7411eaac8d1a9c51bcca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:51.405975 25775 log.cc:824] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:51.406332 25776 tablet_bootstrap.cc:492] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: Bootstrap starting.
I20260928 12:29:51.406849 25775 tablet_bootstrap.cc:492] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: No bootstrap required, opened a new log
I20260928 12:29:51.406940 25775 ts_tablet_manager.cc:1397] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 12:29:51.407023 25776 tablet_bootstrap.cc:654] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:51.407624 25776 log.cc:824] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:51.408419 25776 tablet_bootstrap.cc:492] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: No bootstrap required, opened a new log
I20260928 12:29:51.408491 25776 ts_tablet_manager.cc:1397] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:51.408735 25775 raft_consensus.cc:348] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.408916 25775 raft_consensus.cc:374] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:51.408962 25775 raft_consensus.cc:729] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff10665d9df04ed48ef610c619b1f07f, State: Initialized, Role: FOLLOWER
I20260928 12:29:51.409076 25775 consensus_queue.cc:260] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.409382 25779 tablet_bootstrap.cc:492] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: Bootstrap starting.
I20260928 12:29:51.409464 25775 ts_tablet_manager.cc:1428] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:29:51.409587 25761 heartbeater.cc:503] Master 127.7.19.126:43503 was elected leader, sending a full tablet report...
I20260928 12:29:51.410076 25779 tablet_bootstrap.cc:654] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:51.410363 25776 raft_consensus.cc:348] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.410537 25776 raft_consensus.cc:374] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:51.410584 25776 raft_consensus.cc:729] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9181479154f4f85a480a13e1a8f5d52, State: Initialized, Role: FOLLOWER
I20260928 12:29:51.410689 25779 log.cc:824] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:51.410705 25776 consensus_queue.cc:260] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.410989 25776 ts_tablet_manager.cc:1428] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:29:51.410998 25493 heartbeater.cc:503] Master 127.7.19.126:43503 was elected leader, sending a full tablet report...
I20260928 12:29:51.411676 25779 tablet_bootstrap.cc:492] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: No bootstrap required, opened a new log
I20260928 12:29:51.411751 25779 ts_tablet_manager.cc:1397] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:51.413304 25779 raft_consensus.cc:348] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.413426 25779 raft_consensus.cc:374] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:51.413447 25779 raft_consensus.cc:729] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc0e6d29ebe45e993558ad4ba0264ad, State: Initialized, Role: FOLLOWER
I20260928 12:29:51.413574 25779 consensus_queue.cc:260] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.413851 25779 ts_tablet_manager.cc:1428] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:51.413858 25627 heartbeater.cc:503] Master 127.7.19.126:43503 was elected leader, sending a full tablet report...
W20260928 12:29:51.464560 25628 tablet.cc:2369] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:29:51.557221 25494 tablet.cc:2369] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 12:29:51.613773 25762 tablet.cc:2369] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:29:51.796170 25781 raft_consensus.cc:482] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:51.796309 25781 raft_consensus.cc:504] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.796820 25781 leader_election.cc:288] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dc0e6d29ebe45e993558ad4ba0264ad (127.7.19.66:34095), ff10665d9df04ed48ef610c619b1f07f (127.7.19.67:36273)
I20260928 12:29:51.799867 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.797121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:34647 (local address 127.7.19.66:34095)
0928 12:29:51.797166 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.797169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.797240 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:29:51.797332 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.797333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.797346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.797408 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:51.797656 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.798113 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.799125 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.799494 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.799548 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.799578 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.799735 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.799790 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:29:51.799887 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.797135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:59401 (local address 127.7.19.67:36273)
0928 12:29:51.797182 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.797185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.797248 (+ 63us) server_negotiation.cc:407] Connection header received
0928 12:29:51.797366 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.797367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.797377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.797434 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:51.797629 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.798126 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.799125 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.799501 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.799590 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.799600 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.799783 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.799826 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:29:51.799885 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.797027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:34095 (local address 127.7.19.65:34647)
0928 12:29:51.797175 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.797188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.797257 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.797416 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.797417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.797420 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:51.797502 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.797504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.798195 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.798196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.799024 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.799027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.799581 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.799582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.799654 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.799658 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.799710 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.799768 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260928 12:29:51.799888 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.797029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:36273 (local address 127.7.19.65:59401)
0928 12:29:51.797185 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.797195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:51.797269 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:51.797456 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:51.797457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:51.797459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:51.797552 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.797554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.798232 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.798233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.799025 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:51.799028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.799609 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.799610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:51.799682 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.799686 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:51.799761 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 12:29:51.799800 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260928 12:29:51.800232 25581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07e636211e7411eaac8d1a9c51bcca5" candidate_uuid: "c9181479154f4f85a480a13e1a8f5d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" is_pre_election: true
I20260928 12:29:51.800406 25581 raft_consensus.cc:2454] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9181479154f4f85a480a13e1a8f5d52 in term 0.
I20260928 12:29:51.800522 25715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07e636211e7411eaac8d1a9c51bcca5" candidate_uuid: "c9181479154f4f85a480a13e1a8f5d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff10665d9df04ed48ef610c619b1f07f" is_pre_election: true
I20260928 12:29:51.800704 25715 raft_consensus.cc:2454] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9181479154f4f85a480a13e1a8f5d52 in term 0.
I20260928 12:29:51.800704 25380 leader_election.cc:302] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc0e6d29ebe45e993558ad4ba0264ad, c9181479154f4f85a480a13e1a8f5d52; no voters:
I20260928 12:29:51.800817 25781 raft_consensus.cc:2781] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:51.800881 25781 raft_consensus.cc:482] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:51.800909 25781 raft_consensus.cc:3037] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:51.801628 25781 raft_consensus.cc:504] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.801776 25781 leader_election.cc:288] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [CANDIDATE]: Term 1 election: Requested vote from peers 6dc0e6d29ebe45e993558ad4ba0264ad (127.7.19.66:34095), ff10665d9df04ed48ef610c619b1f07f (127.7.19.67:36273)
I20260928 12:29:51.801949 25715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07e636211e7411eaac8d1a9c51bcca5" candidate_uuid: "c9181479154f4f85a480a13e1a8f5d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff10665d9df04ed48ef610c619b1f07f"
I20260928 12:29:51.801956 25581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07e636211e7411eaac8d1a9c51bcca5" candidate_uuid: "c9181479154f4f85a480a13e1a8f5d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad"
I20260928 12:29:51.802013 25581 raft_consensus.cc:3037] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:51.802032 25715 raft_consensus.cc:3037] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:51.802700 25581 raft_consensus.cc:2454] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9181479154f4f85a480a13e1a8f5d52 in term 1.
I20260928 12:29:51.802876 25715 raft_consensus.cc:2454] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9181479154f4f85a480a13e1a8f5d52 in term 1.
I20260928 12:29:51.802876 25380 leader_election.cc:302] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc0e6d29ebe45e993558ad4ba0264ad, c9181479154f4f85a480a13e1a8f5d52; no voters:
I20260928 12:29:51.803092 25781 raft_consensus.cc:2781] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:51.803256 25781 raft_consensus.cc:686] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 1 LEADER]: Becoming Leader. State: Replica: c9181479154f4f85a480a13e1a8f5d52, State: Running, Role: LEADER
I20260928 12:29:51.803366 25781 consensus_queue.cc:237] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } }
I20260928 12:29:51.804172 25301 catalog_manager.cc:5495] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9181479154f4f85a480a13e1a8f5d52 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "c9181479154f4f85a480a13e1a8f5d52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9181479154f4f85a480a13e1a8f5d52" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 33487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 } health_report { overall_health: UNKNOWN } } }
I20260928 12:29:51.841339 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.838779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.7.19.65:33487)
0928 12:29:51.838853 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:51.838855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:51.838864 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:51.838902 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:51.838903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:51.838923 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:51.838989 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:51.839118 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.839607 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.840407 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:51.840764 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:51.840832 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:51.840958 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:51.841082 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:51.841159 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:51.841222 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 12:29:51.841229 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:29:51.843297 25715 raft_consensus.cc:1261] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 1 FOLLOWER]: Refusing update from remote peer c9181479154f4f85a480a13e1a8f5d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:51.843297 25581 raft_consensus.cc:1261] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 1 FOLLOWER]: Refusing update from remote peer c9181479154f4f85a480a13e1a8f5d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:51.843701 25786 consensus_queue.cc:1035] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff10665d9df04ed48ef610c619b1f07f" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:51.843837 25781 consensus_queue.cc:1035] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc0e6d29ebe45e993558ad4ba0264ad" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:51.846287 25789 mvcc.cc:205] Tried to move back new op lower bound from 7334291832187240448 to 7334291832027168768. Current Snapshot: MvccSnapshot[applied={T|T < 7334291832187240448}]
I20260928 12:29:51.846421 25790 mvcc.cc:205] Tried to move back new op lower bound from 7334291832187240448 to 7334291832027168768. Current Snapshot: MvccSnapshot[applied={T|T < 7334291832187240448}]
I20260928 12:29:51.846422 25791 mvcc.cc:205] Tried to move back new op lower bound from 7334291832187240448 to 7334291832027168768. Current Snapshot: MvccSnapshot[applied={T|T < 7334291832187240448}]
I20260928 12:29:51.848971 25302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:51.849062 25302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:51.849785 25302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 066d23e571cb4cb5b615e7f420f68d5a: Sending DeleteTablet for 3 replicas of tablet b07e636211e7411eaac8d1a9c51bcca5
I20260928 12:29:51.850093 25427 tablet_service.cc:1514] Processing DeleteTablet for tablet b07e636211e7411eaac8d1a9c51bcca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:51 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:48546
I20260928 12:29:51.850126 25695 tablet_service.cc:1514] Processing DeleteTablet for tablet b07e636211e7411eaac8d1a9c51bcca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:51 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:44106
I20260928 12:29:51.850137 25561 tablet_service.cc:1514] Processing DeleteTablet for tablet b07e636211e7411eaac8d1a9c51bcca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:51 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:39166
I20260928 12:29:51.850350 25799 tablet_replica.cc:330] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52: stopping tablet replica
I20260928 12:29:51.850452 25799 raft_consensus.cc:2229] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:29:51.850549 25799 raft_consensus.cc:2258] T b07e636211e7411eaac8d1a9c51bcca5 P c9181479154f4f85a480a13e1a8f5d52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:51.850665 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25364
I20260928 12:29:51.855872 25801 tablet_replica.cc:330] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: stopping tablet replica
I20260928 12:29:51.855934 25800 tablet_replica.cc:330] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad: stopping tablet replica
W20260928 12:29:51.855983 25288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:33487 (error 108)
I20260928 12:29:51.856009 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25498
I20260928 12:29:51.856045 25800 raft_consensus.cc:2229] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:51.856178 25800 raft_consensus.cc:2258] T b07e636211e7411eaac8d1a9c51bcca5 P 6dc0e6d29ebe45e993558ad4ba0264ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:51.856197 25801 raft_consensus.cc:2229] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:51.856375 25801 raft_consensus.cc:2258] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:51.857045 25801 ts_tablet_manager.cc:1905] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:51.857498 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.857347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:33487 (local address 127.0.0.1:48562)
0928 12:29:51.857400 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.857442 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:33487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 12:29:51.857667 25288 catalog_manager.cc:4584] TS c9181479154f4f85a480a13e1a8f5d52 (127.7.19.65:33487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b07e636211e7411eaac8d1a9c51bcca5: Network error: Client connection negotiation failed: client connection to 127.7.19.65:33487: connect: Connection refused (error 111)
I20260928 12:29:51.859302 25801 ts_tablet_manager.cc:1918] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:51.859385 25801 log.cc:1196] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/b07e636211e7411eaac8d1a9c51bcca5
I20260928 12:29:51.859618 25801 ts_tablet_manager.cc:1939] T b07e636211e7411eaac8d1a9c51bcca5 P ff10665d9df04ed48ef610c619b1f07f: Deleting consensus metadata
I20260928 12:29:51.860062 25289 catalog_manager.cc:4841] TS ff10665d9df04ed48ef610c619b1f07f (127.7.19.67:36273): tablet b07e636211e7411eaac8d1a9c51bcca5 (table test-table [id=70f2b5a5e0b248d4949f9319e241e47d]) successfully deleted
I20260928 12:29:51.861850 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:51.861722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:34095 (local address 127.0.0.1:39168)
0928 12:29:51.861784 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:51.861817 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.66:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 12:29:51.861855 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25632
I20260928 12:29:51.866432 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25270
2026-09-28T12:29:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2289 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T12:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:51Z Disabled control of system clock
I20260928 12:29:51.893697 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41443
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:51.974848 25806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:51.975062 25806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:51.975085 25806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:51.976163 25806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:51.976197 25806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:51.976210 25806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:51.976230 25806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:51.976243 25806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:51.977547 25806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:41443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:51.977813 25806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:51.978055 25806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:51.982848 25814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:51.982969 25806 server_base.cc:1034] running on GCE node
W20260928 12:29:51.982857 25812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:51.983105 25811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:51.983377 25806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:51.983626 25806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:51.984817 25806 hybrid_clock.cc:648] HybridClock initialized: now 1790598591984808 us; error 33 us; skew 500 ppm
I20260928 12:29:51.990044 25806 webserver.cc:460] Webserver started at http://127.7.19.126:39845/ using document root <none> and password file <none>
I20260928 12:29:51.990317 25806 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:51.990370 25806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:51.990535 25806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:51.991627 25806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "38f281661675448e940d7795e275d8cd"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "83adfdf0bfd3e6a23ad8c781b0ea3ab5"
server_key_iv: "57a495ebd80248635cabdf35639a558c"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.992002 25806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "38f281661675448e940d7795e275d8cd"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "83adfdf0bfd3e6a23ad8c781b0ea3ab5"
server_key_iv: "57a495ebd80248635cabdf35639a558c"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.993408 25806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:51.994175 25820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:51.994386 25806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:51.994482 25806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "38f281661675448e940d7795e275d8cd"
format_stamp: "Formatted at 2026-09-28 12:29:51 on dist-test-slave-cb7x"
server_key: "83adfdf0bfd3e6a23ad8c781b0ea3ab5"
server_key_iv: "57a495ebd80248635cabdf35639a558c"
server_key_version: "encryptionkey@0"
I20260928 12:29:51.994554 25806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:52.008051 25806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:52.008411 25806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:52.008571 25806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:52.012279 25806 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:41443
I20260928 12:29:52.012348 25872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:41443 every 8 connection(s)
I20260928 12:29:52.012756 25806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:52.013737 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:52.015969 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd: Bootstrap starting.
I20260928 12:29:52.016561 25873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:52.016893 25873 log.cc:824] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:52.017597 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd: No bootstrap required, opened a new log
I20260928 12:29:52.018326 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25806
I20260928 12:29:52.018399 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:52.018612 7245 external_mini_cluster.cc:1428] Setting key a987d7da95f9cc8810f2edab9ac0109f
I20260928 12:29:52.019742 25873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f281661675448e940d7795e275d8cd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41443 } }
I20260928 12:29:52.019913 25873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:52.019950 25873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f281661675448e940d7795e275d8cd, State: Initialized, Role: FOLLOWER
I20260928 12:29:52.020095 25873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f281661675448e940d7795e275d8cd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41443 } }
I20260928 12:29:52.020211 25873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:52.020246 25873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:52.020294 25873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:52.021039 25873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f281661675448e940d7795e275d8cd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41443 } }
I20260928 12:29:52.021150 25873 leader_election.cc:302] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38f281661675448e940d7795e275d8cd; no voters:
I20260928 12:29:52.021323 25873 leader_election.cc:288] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:52.021402 25878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:52.021548 25878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [term 1 LEADER]: Becoming Leader. State: Replica: 38f281661675448e940d7795e275d8cd, State: Running, Role: LEADER
I20260928 12:29:52.021564 25873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:52.021633 25878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f281661675448e940d7795e275d8cd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41443 } }
I20260928 12:29:52.022035 25879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38f281661675448e940d7795e275d8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f281661675448e940d7795e275d8cd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41443 } } }
I20260928 12:29:52.022027 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f281661675448e940d7795e275d8cd. Latest consensus state: current_term: 1 leader_uuid: "38f281661675448e940d7795e275d8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38f281661675448e940d7795e275d8cd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41443 } } }
I20260928 12:29:52.022198 25879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:52.022209 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:52.022979 25887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:52.023694 25887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:52.025490 25887 catalog_manager.cc:1347] Generated new cluster ID: 5ebfc41bcaf84397a52d1be28c6afe38
I20260928 12:29:52.025547 25887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:52.025764 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.019140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.7.19.126:41443)
0928 12:29:52.019366 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.019376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.019411 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:29:52.019712 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.019720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.019798 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.019950 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:52.020210 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.022055 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.023278 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.023707 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.024174 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.025049 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:52.025053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:52.025054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:52.025237 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:52.025277 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:52.025288 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:52.025377 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.025455 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.025537 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260928 12:29:52.040712 25887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:52.041296 25887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:52.046260 25887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd: Generated new TSK 0
I20260928 12:29:52.046471 25887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:29:52.049940 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41443
--builtin_ntp_servers=127.7.19.84:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:52.133275 25897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:52.133489 25897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:52.133512 25897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:52.134778 25897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:52.134816 25897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:52.134843 25897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:52.136211 25897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:52.136467 25897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:52.136857 25897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:52.141767 25903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:52.141744 25902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:52.141743 25905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:52.142377 25897 server_base.cc:1034] running on GCE node
I20260928 12:29:52.142570 25897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:52.142818 25897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:52.143993 25897 hybrid_clock.cc:648] HybridClock initialized: now 1790598592143978 us; error 28 us; skew 500 ppm
I20260928 12:29:52.145394 25897 webserver.cc:460] Webserver started at http://127.7.19.65:39175/ using document root <none> and password file <none>
I20260928 12:29:52.145671 25897 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:52.145746 25897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:52.145900 25897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:52.146970 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "db7a89ee2b874ba795152d88d6d96a91"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "f6b7aceebb261ae9fb60dd37532a7278"
server_key_iv: "07f05968c10203abc7054e228ec39d27"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.147346 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "db7a89ee2b874ba795152d88d6d96a91"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "f6b7aceebb261ae9fb60dd37532a7278"
server_key_iv: "07f05968c10203abc7054e228ec39d27"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.148802 25897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:29:52.149601 25911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.149796 25897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:52.149893 25897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "db7a89ee2b874ba795152d88d6d96a91"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "f6b7aceebb261ae9fb60dd37532a7278"
server_key_iv: "07f05968c10203abc7054e228ec39d27"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.149983 25897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:52.171566 25897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:52.171900 25897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:52.172063 25897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:52.172315 25897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:52.172643 25897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:52.172750 25897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.172794 25897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:52.172820 25897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.178804 25897 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:45167
I20260928 12:29:52.178900 26024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:45167 every 8 connection(s)
I20260928 12:29:52.179270 25897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:52.184530 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.180012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:58457 (local address 127.7.19.126:41443)
0928 12:29:52.180082 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.180084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.180335 (+ 251us) server_negotiation.cc:407] Connection header received
0928 12:29:52.180930 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.180932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.180948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.181001 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:52.181673 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.182181 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.183165 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.183372 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.183450 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.184247 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:52.184248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:52.184248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:52.184253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:52.184268 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:52.184272 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:52.184324 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.184420 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.184450 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:29:52.184777 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.180012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41443 (local address 127.7.19.65:58457)
0928 12:29:52.180214 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.180246 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.180822 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.181078 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.181082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.181099 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:52.181577 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.181581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.182276 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.182277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.183072 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.183074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.183456 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.183458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.183591 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.184109 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:52.184110 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:52.184138 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:52.184160 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:52.184331 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:52.184335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:52.184338 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.184421 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.184517 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260928 12:29:52.185184 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 25897
I20260928 12:29:52.185310 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:29:52.185325 26025 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41443
I20260928 12:29:52.185416 26025 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:52.185514 7245 external_mini_cluster.cc:1428] Setting key dc9d86c4910c30c3d14af71d79005852
I20260928 12:29:52.185624 26025 heartbeater.cc:511] Master 127.7.19.126:41443 requested a full tablet report, sending...
I20260928 12:29:52.186137 25836 ts_manager.cc:194] Registered new tserver with Master: db7a89ee2b874ba795152d88d6d96a91 (127.7.19.65:45167)
I20260928 12:29:52.186439 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41443
--builtin_ntp_servers=127.7.19.84:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:29:52.186769 25836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.65:58457
W20260928 12:29:52.268532 26028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:52.268812 26028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:52.268841 26028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:52.269968 26028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:52.270004 26028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:52.270030 26028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:52.271317 26028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:52.271510 26028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:52.271759 26028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:52.276855 26036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:52.276873 26028 server_base.cc:1034] running on GCE node
W20260928 12:29:52.276830 26034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:52.276813 26033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:52.277254 26028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:52.277462 26028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:52.278609 26028 hybrid_clock.cc:648] HybridClock initialized: now 1790598592278591 us; error 41 us; skew 500 ppm
I20260928 12:29:52.279909 26028 webserver.cc:460] Webserver started at http://127.7.19.66:41787/ using document root <none> and password file <none>
I20260928 12:29:52.280114 26028 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:52.280154 26028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:52.280243 26028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:52.281301 26028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "adbe10090d2045d99cb533742023bb49"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "04c98e5e74291d993411e165ae1f7c0a"
server_key_iv: "558ed6c4360effdd4eecc191b762e89b"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.281594 26028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "adbe10090d2045d99cb533742023bb49"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "04c98e5e74291d993411e165ae1f7c0a"
server_key_iv: "558ed6c4360effdd4eecc191b762e89b"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.282826 26028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:29:52.283470 26042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.283703 26028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:52.283793 26028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "adbe10090d2045d99cb533742023bb49"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "04c98e5e74291d993411e165ae1f7c0a"
server_key_iv: "558ed6c4360effdd4eecc191b762e89b"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.283890 26028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:52.291970 26028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:52.292254 26028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:52.292405 26028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:52.292634 26028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:52.292997 26028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:52.293035 26028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.293056 26028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:52.293108 26028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.299149 26028 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:35815
I20260928 12:29:52.299217 26155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:35815 every 8 connection(s)
I20260928 12:29:52.299554 26028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:29:52.300904 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26028
I20260928 12:29:52.301030 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:29:52.301225 7245 external_mini_cluster.cc:1428] Setting key 2ee3a4745e0337b31e3bcb4f84355620
I20260928 12:29:52.302309 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41443
--builtin_ntp_servers=127.7.19.84:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:29:52.305543 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.300273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49083 (local address 127.7.19.126:41443)
0928 12:29:52.300339 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.300341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.300593 (+ 252us) server_negotiation.cc:407] Connection header received
0928 12:29:52.301322 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.301323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.301336 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.301406 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:52.302250 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.302892 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.303973 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.304204 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.304297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.305231 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:52.305233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:52.305233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:52.305238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:52.305257 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:52.305261 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:52.305334 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.305404 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.305435 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 12:29:52.305742 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.300253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41443 (local address 127.7.19.66:49083)
0928 12:29:52.300469 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.300503 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.301205 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.301483 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.301487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.301499 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:52.302153 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.302159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.302991 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.302992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.303859 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.303861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.304318 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.304319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.304467 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.305089 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:52.305090 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:52.305117 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:52.305138 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:52.305333 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:52.305335 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:52.305338 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.305384 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.305462 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260928 12:29:52.306166 26156 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41443
I20260928 12:29:52.306255 26156 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:52.306448 26156 heartbeater.cc:511] Master 127.7.19.126:41443 requested a full tablet report, sending...
I20260928 12:29:52.306795 25836 ts_manager.cc:194] Registered new tserver with Master: adbe10090d2045d99cb533742023bb49 (127.7.19.66:35815)
I20260928 12:29:52.307173 25836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.66:49083
W20260928 12:29:52.386804 26159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:52.387025 26159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:52.387059 26159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:52.388236 26159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:52.388298 26159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:52.388339 26159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:52.389679 26159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:52.389935 26159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:52.390224 26159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:52.395319 26165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:52.395335 26167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:52.395323 26164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:52.395339 26159 server_base.cc:1034] running on GCE node
I20260928 12:29:52.395671 26159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:52.395869 26159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:52.397002 26159 hybrid_clock.cc:648] HybridClock initialized: now 1790598592396970 us; error 53 us; skew 500 ppm
I20260928 12:29:52.398456 26159 webserver.cc:460] Webserver started at http://127.7.19.67:45129/ using document root <none> and password file <none>
I20260928 12:29:52.398649 26159 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:52.398691 26159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:52.398778 26159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:52.399771 26159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "e7dba953a5e44e13ac61673c57cbe90e"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "7c426c36b04a8c0927456f720a6fb9a1"
server_key_iv: "39b2e5259e5723d26b1d68fb9ca39cc3"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.400063 26159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "e7dba953a5e44e13ac61673c57cbe90e"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "7c426c36b04a8c0927456f720a6fb9a1"
server_key_iv: "39b2e5259e5723d26b1d68fb9ca39cc3"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.401422 26159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:29:52.402230 26173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.402545 26159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:52.402648 26159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "e7dba953a5e44e13ac61673c57cbe90e"
format_stamp: "Formatted at 2026-09-28 12:29:52 on dist-test-slave-cb7x"
server_key: "7c426c36b04a8c0927456f720a6fb9a1"
server_key_iv: "39b2e5259e5723d26b1d68fb9ca39cc3"
server_key_version: "encryptionkey@0"
I20260928 12:29:52.402726 26159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:52.412407 26159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:52.412753 26159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:52.412909 26159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:52.413131 26159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:52.413457 26159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:52.413513 26159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.413554 26159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:52.413581 26159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:52.419469 26159 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:45665
I20260928 12:29:52.419555 26286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:45665 every 8 connection(s)
I20260928 12:29:52.419893 26159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:29:52.425274 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.420735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:55045 (local address 127.7.19.126:41443)
0928 12:29:52.420806 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.420813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.421118 (+ 305us) server_negotiation.cc:407] Connection header received
0928 12:29:52.421690 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.421692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.421703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.421786 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:52.422577 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.423070 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.424044 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.424232 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.424286 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.425073 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:52.425074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:52.425075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:52.425079 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:52.425091 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:52.425095 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:52.425142 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.425192 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.425216 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:29:52.425475 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.420721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41443 (local address 127.7.19.67:55045)
0928 12:29:52.420999 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.421034 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.421567 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.421843 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.421846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.421854 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:52.422448 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.422455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.423192 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.423193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.423967 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.423969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.424311 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.424312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.424417 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.424945 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:52.424945 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:52.424973 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:52.424995 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:52.425145 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:52.425147 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:52.425149 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.425193 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.425267 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260928 12:29:52.426021 26287 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41443
I20260928 12:29:52.426111 26287 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:52.426245 26287 heartbeater.cc:511] Master 127.7.19.126:41443 requested a full tablet report, sending...
I20260928 12:29:52.426605 25836 ts_manager.cc:194] Registered new tserver with Master: e7dba953a5e44e13ac61673c57cbe90e (127.7.19.67:45665)
I20260928 12:29:52.426927 25836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.67:55045
I20260928 12:29:52.427171 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26159
I20260928 12:29:52.427304 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:29:52.427503 7245 external_mini_cluster.cc:1428] Setting key 5668461c9a60a6230d6f45582045938b
I20260928 12:29:52.428314 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:52.432235 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.429634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.7.19.126:41443)
0928 12:29:52.429676 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.429678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.429828 (+ 150us) server_negotiation.cc:407] Connection header received
0928 12:29:52.429881 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.429882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.429890 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.429931 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:52.430050 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.430474 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.431198 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.431367 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.431420 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.431989 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:52.431990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:52.431991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:52.431994 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:52.432006 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:52.432009 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:52.432070 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.432110 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.432118 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:29:52.433713 25836 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:52.435210 25836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:52.441373 26295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.437233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:45665 (local address 127.0.0.1:56402)
0928 12:29:52.437333 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.437372 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.437432 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.437945 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.437949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.437953 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.438020 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.438021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.438972 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.438973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.440166 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.440169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.441053 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.441054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.441129 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.441141 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.441245 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.441306 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":56,"threads_started":1}
I20260928 12:29:52.441382 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.437607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.7.19.67:45665)
0928 12:29:52.437755 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.437759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.437770 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:29:52.437813 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.437814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.437830 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.438103 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.438159 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.438884 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.440263 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.440953 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.441049 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.441140 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.441262 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.441299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260928 12:29:52.441582 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.437442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:35815 (local address 127.0.0.1:34388)
0928 12:29:52.437571 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.437581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.437623 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.437996 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.437997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.437999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.438053 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.438055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.439369 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.439370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.440446 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.440448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.441325 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.441326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.441376 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.441379 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.441408 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.441451 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":24,"thread_start_us":92,"threads_started":1}
I20260928 12:29:52.441659 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.437255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:45167 (local address 127.0.0.1:36120)
0928 12:29:52.437546 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.437556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.437638 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.438089 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.438090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.438092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.438125 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.438127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.439389 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.439390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.440264 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.440267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.441292 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.441293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.441356 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.441360 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.441408 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.441451 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260928 12:29:52.441684 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.437350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.7.19.65:45167)
0928 12:29:52.437771 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.437774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.437782 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:29:52.437814 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.437815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.437827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.437911 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.438210 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.439304 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.440349 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.441099 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.441172 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.441254 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.441561 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.441602 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":79,"threads_started":1}
I20260928 12:29:52.441745 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.437553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.7.19.66:35815)
0928 12:29:52.437826 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.437830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.437839 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:52.437877 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.437879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.437894 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.437990 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.438153 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.439273 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.440530 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.441241 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.441493 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.441597 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.441621 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.441658 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":84,"threads_started":1}
I20260928 12:29:52.442380 26090 tablet_service.cc:1467] Processing CreateTablet for tablet 24c15b4dad0b48a2ac68e5856729d595 (DEFAULT_TABLE table=test-table [id=4aff06d4524e49a98a32350c3fd8b874]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:52.442380 25959 tablet_service.cc:1467] Processing CreateTablet for tablet 24c15b4dad0b48a2ac68e5856729d595 (DEFAULT_TABLE table=test-table [id=4aff06d4524e49a98a32350c3fd8b874]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:52.442745 26090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c15b4dad0b48a2ac68e5856729d595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:52.442745 25959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c15b4dad0b48a2ac68e5856729d595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:52.442822 26221 tablet_service.cc:1467] Processing CreateTablet for tablet 24c15b4dad0b48a2ac68e5856729d595 (DEFAULT_TABLE table=test-table [id=4aff06d4524e49a98a32350c3fd8b874]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:52.443163 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c15b4dad0b48a2ac68e5856729d595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:52.445814 26301 tablet_bootstrap.cc:492] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Bootstrap starting.
I20260928 12:29:52.446094 26302 tablet_bootstrap.cc:492] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91: Bootstrap starting.
I20260928 12:29:52.446364 26303 tablet_bootstrap.cc:492] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Bootstrap starting.
I20260928 12:29:52.446511 26301 tablet_bootstrap.cc:654] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:52.446852 26301 log.cc:824] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:52.446884 26302 tablet_bootstrap.cc:654] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:52.446987 26303 tablet_bootstrap.cc:654] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:52.447192 26302 log.cc:824] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:52.447294 26303 log.cc:824] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:52.447599 26301 tablet_bootstrap.cc:492] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: No bootstrap required, opened a new log
I20260928 12:29:52.447677 26301 ts_tablet_manager.cc:1397] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:52.448078 26302 tablet_bootstrap.cc:492] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91: No bootstrap required, opened a new log
I20260928 12:29:52.448150 26302 ts_tablet_manager.cc:1397] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:52.448975 26303 tablet_bootstrap.cc:492] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: No bootstrap required, opened a new log
I20260928 12:29:52.449064 26303 ts_tablet_manager.cc:1397] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 12:29:52.450361 26301 raft_consensus.cc:348] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.450539 26301 raft_consensus.cc:374] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:52.450574 26301 raft_consensus.cc:729] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7dba953a5e44e13ac61673c57cbe90e, State: Initialized, Role: FOLLOWER
I20260928 12:29:52.450706 26301 consensus_queue.cc:260] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.450755 26302 raft_consensus.cc:348] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.450917 26302 raft_consensus.cc:374] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:52.450959 26302 raft_consensus.cc:729] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7a89ee2b874ba795152d88d6d96a91, State: Initialized, Role: FOLLOWER
I20260928 12:29:52.450982 26301 ts_tablet_manager.cc:1428] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 12:29:52.451085 26287 heartbeater.cc:503] Master 127.7.19.126:41443 was elected leader, sending a full tablet report...
I20260928 12:29:52.451073 26302 consensus_queue.cc:260] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.451324 26302 ts_tablet_manager.cc:1428] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:29:52.451237 26303 raft_consensus.cc:348] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.451388 26303 raft_consensus.cc:374] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:52.451421 26303 raft_consensus.cc:729] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbe10090d2045d99cb533742023bb49, State: Initialized, Role: FOLLOWER
I20260928 12:29:52.451444 26025 heartbeater.cc:503] Master 127.7.19.126:41443 was elected leader, sending a full tablet report...
I20260928 12:29:52.451541 26303 consensus_queue.cc:260] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.451817 26303 ts_tablet_manager.cc:1428] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:29:52.451936 26156 heartbeater.cc:503] Master 127.7.19.126:41443 was elected leader, sending a full tablet report...
I20260928 12:29:52.473963 26309 raft_consensus.cc:482] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:52.474025 26307 raft_consensus.cc:482] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:52.474112 26307 raft_consensus.cc:504] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.474112 26309 raft_consensus.cc:504] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.474507 26309 leader_election.cc:288] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7dba953a5e44e13ac61673c57cbe90e (127.7.19.67:45665), db7a89ee2b874ba795152d88d6d96a91 (127.7.19.65:45167)
I20260928 12:29:52.474509 26307 leader_election.cc:288] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7a89ee2b874ba795152d88d6d96a91 (127.7.19.65:45167), adbe10090d2045d99cb533742023bb49 (127.7.19.66:35815)
I20260928 12:29:52.478467 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.475024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:60679 (local address 127.7.19.66:35815)
0928 12:29:52.475062 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.475063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.475107 (+ 44us) server_negotiation.cc:407] Connection header received
0928 12:29:52.475199 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.475200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.475208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.475235 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.475434 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476132 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.477605 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478108 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478150 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478159 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.478360 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.478395 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 12:29:52.478538 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:39459 (local address 127.7.19.67:45665)
0928 12:29:52.474971 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.474973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.475171 (+ 198us) server_negotiation.cc:407] Connection header received
0928 12:29:52.475279 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.475280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.475287 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.475317 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.475526 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476206 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.477621 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478139 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478181 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478190 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.478421 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.478459 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:29:52.478538 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:45167 (local address 127.7.19.67:34315)
0928 12:29:52.474899 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.474910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.474958 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.475220 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.475221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.475223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.475291 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.475294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476295 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.476296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.477493 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.477495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478335 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.478406 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478409 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.478438 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.478471 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260928 12:29:52.478742 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:34315 (local address 127.7.19.65:45167)
0928 12:29:52.475070 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.475072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.475080 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:29:52.475105 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.475106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.475116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.475358 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.475393 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476192 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.477615 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478236 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478581 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478593 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.478612 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.478656 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260928 12:29:52.478818 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:45665 (local address 127.7.19.66:39459)
0928 12:29:52.475118 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.475128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.475195 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.475374 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.475375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.475377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.475451 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.475454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476304 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.476305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.477509 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.477512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478251 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.478340 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478344 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.478694 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.478733 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":116,"threads_started":1}
I20260928 12:29:52.478919 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:35815 (local address 127.7.19.67:60679)
0928 12:29:52.475055 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.475063 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.475130 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.475291 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.475291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.475293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.475361 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.475362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476230 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.476231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.477483 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.477485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478214 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.478286 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478290 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.478554 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.478662 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":113,"threads_started":1}
I20260928 12:29:52.479017 25978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "e7dba953a5e44e13ac61673c57cbe90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7a89ee2b874ba795152d88d6d96a91" is_pre_election: true
I20260928 12:29:52.479058 26241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "adbe10090d2045d99cb533742023bb49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dba953a5e44e13ac61673c57cbe90e" is_pre_election: true
I20260928 12:29:52.479143 26241 raft_consensus.cc:2454] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adbe10090d2045d99cb533742023bb49 in term 0.
I20260928 12:29:52.479173 25978 raft_consensus.cc:2454] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dba953a5e44e13ac61673c57cbe90e in term 0.
I20260928 12:29:52.479336 26110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "e7dba953a5e44e13ac61673c57cbe90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbe10090d2045d99cb533742023bb49" is_pre_election: true
I20260928 12:29:52.479395 26046 leader_election.cc:302] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbe10090d2045d99cb533742023bb49, e7dba953a5e44e13ac61673c57cbe90e; no voters:
I20260928 12:29:52.479457 26110 raft_consensus.cc:2454] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7dba953a5e44e13ac61673c57cbe90e in term 0.
I20260928 12:29:52.479489 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:54009 (local address 127.7.19.65:45167)
0928 12:29:52.475001 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.475003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.475012 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:52.475041 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.475042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.475054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.475106 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:52.475361 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476129 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.477982 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.478509 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.478569 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.478577 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.479364 (+ 787us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.479410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 12:29:52.479509 26309 raft_consensus.cc:2781] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:52.479554 26309 raft_consensus.cc:482] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:52.479581 26309 raft_consensus.cc:3037] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:52.479638 26177 leader_election.cc:302] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbe10090d2045d99cb533742023bb49, e7dba953a5e44e13ac61673c57cbe90e; no voters:
I20260928 12:29:52.479760 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.474679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:45167 (local address 127.7.19.66:54009)
0928 12:29:52.474883 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.474896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:52.474947 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:52.475181 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:52.475182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:52.475184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:52.475277 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.475280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.476234 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.476235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.477481 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:52.477484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.479137 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.479138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:52.479205 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.479209 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:52.479256 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 12:29:52.479298 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 12:29:52.479849 26307 raft_consensus.cc:2781] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:52.479900 26307 raft_consensus.cc:482] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:52.479924 26307 raft_consensus.cc:3037] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:52.480019 25978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "adbe10090d2045d99cb533742023bb49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7a89ee2b874ba795152d88d6d96a91" is_pre_election: true
I20260928 12:29:52.480085 25978 raft_consensus.cc:2454] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adbe10090d2045d99cb533742023bb49 in term 0.
I20260928 12:29:52.480587 26309 raft_consensus.cc:504] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.480784 26307 raft_consensus.cc:504] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.480870 26241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "adbe10090d2045d99cb533742023bb49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7dba953a5e44e13ac61673c57cbe90e"
I20260928 12:29:52.480917 26307 leader_election.cc:288] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [CANDIDATE]: Term 1 election: Requested vote from peers db7a89ee2b874ba795152d88d6d96a91 (127.7.19.65:45167), adbe10090d2045d99cb533742023bb49 (127.7.19.66:35815)
I20260928 12:29:52.480962 26241 raft_consensus.cc:2379] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate adbe10090d2045d99cb533742023bb49 in current term 1: Already voted for candidate e7dba953a5e44e13ac61673c57cbe90e in this term.
I20260928 12:29:52.481019 25978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "adbe10090d2045d99cb533742023bb49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7a89ee2b874ba795152d88d6d96a91"
I20260928 12:29:52.481043 25979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "e7dba953a5e44e13ac61673c57cbe90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7a89ee2b874ba795152d88d6d96a91"
I20260928 12:29:52.481071 25978 raft_consensus.cc:3037] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:52.481086 26110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c15b4dad0b48a2ac68e5856729d595" candidate_uuid: "e7dba953a5e44e13ac61673c57cbe90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbe10090d2045d99cb533742023bb49"
I20260928 12:29:52.481159 26110 raft_consensus.cc:2379] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7dba953a5e44e13ac61673c57cbe90e in current term 1: Already voted for candidate adbe10090d2045d99cb533742023bb49 in this term.
I20260928 12:29:52.481246 26309 leader_election.cc:288] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [CANDIDATE]: Term 1 election: Requested vote from peers e7dba953a5e44e13ac61673c57cbe90e (127.7.19.67:45665), db7a89ee2b874ba795152d88d6d96a91 (127.7.19.65:45167)
I20260928 12:29:52.481389 26175 leader_election.cc:302] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7dba953a5e44e13ac61673c57cbe90e; no voters: adbe10090d2045d99cb533742023bb49, db7a89ee2b874ba795152d88d6d96a91
I20260928 12:29:52.481552 26307 raft_consensus.cc:2726] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 12:29:52.481905 25978 raft_consensus.cc:2454] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adbe10090d2045d99cb533742023bb49 in term 1.
I20260928 12:29:52.482110 26044 leader_election.cc:302] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: adbe10090d2045d99cb533742023bb49, db7a89ee2b874ba795152d88d6d96a91; no voters: e7dba953a5e44e13ac61673c57cbe90e
I20260928 12:29:52.482236 26309 raft_consensus.cc:2781] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:52.482383 26309 raft_consensus.cc:686] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 1 LEADER]: Becoming Leader. State: Replica: adbe10090d2045d99cb533742023bb49, State: Running, Role: LEADER
I20260928 12:29:52.482479 26309 consensus_queue.cc:237] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } }
I20260928 12:29:52.483347 25837 catalog_manager.cc:5495] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 reported cstate change: term changed from 0 to 1, leader changed from <none> to adbe10090d2045d99cb533742023bb49 (127.7.19.66). New cstate: current_term: 1 leader_uuid: "adbe10090d2045d99cb533742023bb49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adbe10090d2045d99cb533742023bb49" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 35815 } health_report { overall_health: HEALTHY } } }
I20260928 12:29:52.491714 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.489418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.7.19.66:35815)
0928 12:29:52.489458 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:52.489460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:52.489471 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:29:52.489525 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:52.489525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:52.489539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:52.489611 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:52.489747 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.490199 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.490951 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:52.491136 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:52.491206 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:52.491368 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:52.491463 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:52.491573 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:52.491603 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 12:29:52.491612 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:29:52.493471 26241 raft_consensus.cc:1261] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 1 FOLLOWER]: Refusing update from remote peer adbe10090d2045d99cb533742023bb49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:52.493471 25978 raft_consensus.cc:1261] T 24c15b4dad0b48a2ac68e5856729d595 P db7a89ee2b874ba795152d88d6d96a91 [term 1 FOLLOWER]: Refusing update from remote peer adbe10090d2045d99cb533742023bb49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:52.493760 26313 consensus_queue.cc:1035] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7a89ee2b874ba795152d88d6d96a91" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:52.493875 26309 consensus_queue.cc:1035] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7dba953a5e44e13ac61673c57cbe90e" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 45665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:52.496737 26318 mvcc.cc:205] Tried to move back new op lower bound from 7334291834850607104 to 7334291834808881152. Current Snapshot: MvccSnapshot[applied={T|T < 7334291834850607104 or (T in {7334291834850607104})}]
I20260928 12:29:52.496930 26317 mvcc.cc:205] Tried to move back new op lower bound from 7334291834850607104 to 7334291834808881152. Current Snapshot: MvccSnapshot[applied={T|T < 7334291834850607104}]
I20260928 12:29:52.498814 25837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:52.498894 25837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:52.499634 25837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38f281661675448e940d7795e275d8cd: Sending DeleteTablet for 3 replicas of tablet 24c15b4dad0b48a2ac68e5856729d595
I20260928 12:29:52.499951 25959 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c15b4dad0b48a2ac68e5856729d595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:52 UTC) from {username='slave'} at 127.0.0.1:36120
I20260928 12:29:52.500020 26090 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c15b4dad0b48a2ac68e5856729d595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:52 UTC) from {username='slave'} at 127.0.0.1:34388
I20260928 12:29:52.499984 26221 tablet_service.cc:1514] Processing DeleteTablet for tablet 24c15b4dad0b48a2ac68e5856729d595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:52 UTC) from {username='slave'} at 127.0.0.1:56402
I20260928 12:29:52.500213 26327 tablet_replica.cc:330] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: stopping tablet replica
I20260928 12:29:52.500245 26328 tablet_replica.cc:330] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: stopping tablet replica
I20260928 12:29:52.500311 26327 raft_consensus.cc:2229] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 12:29:52.500425 26327 raft_consensus.cc:2258] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:52.500443 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25897
I20260928 12:29:52.500442 26328 raft_consensus.cc:2229] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:52.500725 26328 raft_consensus.cc:2258] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:52.501497 26328 ts_tablet_manager.cc:1905] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:52.501891 26327 ts_tablet_manager.cc:1905] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:52.503353 26328 ts_tablet_manager.cc:1918] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:52.503497 26328 log.cc:1196] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/24c15b4dad0b48a2ac68e5856729d595
I20260928 12:29:52.503852 26328 ts_tablet_manager.cc:1939] T 24c15b4dad0b48a2ac68e5856729d595 P e7dba953a5e44e13ac61673c57cbe90e: Deleting consensus metadata
I20260928 12:29:52.503849 26327 ts_tablet_manager.cc:1918] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:52.503912 26327 log.cc:1196] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/24c15b4dad0b48a2ac68e5856729d595
I20260928 12:29:52.504154 26327 ts_tablet_manager.cc:1939] T 24c15b4dad0b48a2ac68e5856729d595 P adbe10090d2045d99cb533742023bb49: Deleting consensus metadata
I20260928 12:29:52.504448 25824 catalog_manager.cc:4841] TS e7dba953a5e44e13ac61673c57cbe90e (127.7.19.67:45665): tablet 24c15b4dad0b48a2ac68e5856729d595 (table test-table [id=4aff06d4524e49a98a32350c3fd8b874]) successfully deleted
I20260928 12:29:52.504738 25824 catalog_manager.cc:4841] TS adbe10090d2045d99cb533742023bb49 (127.7.19.66:35815): tablet 24c15b4dad0b48a2ac68e5856729d595 (table test-table [id=4aff06d4524e49a98a32350c3fd8b874]) successfully deleted
W20260928 12:29:52.507407 25822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:45167 (error 108)
I20260928 12:29:52.507663 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26028
I20260928 12:29:52.508625 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:52.508487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:45167 (local address 127.0.0.1:36122)
0928 12:29:52.508545 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:52.508588 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 12:29:52.508771 25822 catalog_manager.cc:4584] TS db7a89ee2b874ba795152d88d6d96a91 (127.7.19.65:45167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24c15b4dad0b48a2ac68e5856729d595: Network error: Client connection negotiation failed: client connection to 127.7.19.65:45167: connect: Connection refused (error 111)
I20260928 12:29:52.512918 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26159
I20260928 12:29:52.517674 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 25806
2026-09-28T12:29:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (650 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:52 dist-test-slave-cb7x krb5kdc[26332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:52 dist-test-slave-cb7x krb5kdc[26332](info): set up 2 sockets
Sep 28 12:29:52 dist-test-slave-cb7x krb5kdc[26332](info): commencing operation
krb5kdc: starting...
W20260928 12:29:53.552573 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:29:53 dist-test-slave-cb7x krb5kdc[26332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:29:53.672257 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45111
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:53.752250 26345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:53.752458 26345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:53.752480 26345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:53.753677 26345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:53.753719 26345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:53.753734 26345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:53.753747 26345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:53.753759 26345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:53.755126 26345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:53.755316 26345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:53.755478 26345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:53.760430 26350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:53.760535 26353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:53.760579 26351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:53.760746 26345 server_base.cc:1034] running on GCE node
I20260928 12:29:53.760972 26345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:53.761210 26345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:53.762409 26345 hybrid_clock.cc:648] HybridClock initialized: now 1790598593762392 us; error 26 us; skew 500 ppm
Sep 28 12:29:53 dist-test-slave-cb7x krb5kdc[26332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598593, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:53.765003 26345 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:29:53.765486 26345 webserver.cc:460] Webserver started at http://127.7.19.126:46239/ using document root <none> and password file <none>
I20260928 12:29:53.765724 26345 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:53.765806 26345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:53.765969 26345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:53.767100 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "12fdbbff3d2c47fb961045f7a6d48384"
format_stamp: "Formatted at 2026-09-28 12:29:53 on dist-test-slave-cb7x"
server_key: "61a629250f97725103f891efb802cce7"
server_key_iv: "1fed2673dbcbf54fd438a37398c7733b"
server_key_version: "encryptionkey@0"
I20260928 12:29:53.767489 26345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "12fdbbff3d2c47fb961045f7a6d48384"
format_stamp: "Formatted at 2026-09-28 12:29:53 on dist-test-slave-cb7x"
server_key: "61a629250f97725103f891efb802cce7"
server_key_iv: "1fed2673dbcbf54fd438a37398c7733b"
server_key_version: "encryptionkey@0"
I20260928 12:29:53.768980 26345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:53.769809 26360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:53.770004 26345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:53.770115 26345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "12fdbbff3d2c47fb961045f7a6d48384"
format_stamp: "Formatted at 2026-09-28 12:29:53 on dist-test-slave-cb7x"
server_key: "61a629250f97725103f891efb802cce7"
server_key_iv: "1fed2673dbcbf54fd438a37398c7733b"
server_key_version: "encryptionkey@0"
I20260928 12:29:53.770195 26345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:53.795491 26345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:53.797756 26345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:53.797940 26345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:53.802239 26345 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:45111
I20260928 12:29:53.802274 26412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:45111 every 8 connection(s)
I20260928 12:29:53.802645 26345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:53.803617 26413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:53.805895 26413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384: Bootstrap starting.
I20260928 12:29:53.806437 26413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:53.806717 26413 log.cc:824] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:53.806905 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26345
I20260928 12:29:53.807011 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:53.807224 7245 external_mini_cluster.cc:1428] Setting key 4b8c030f25bd587b29d2bbc59228e6cd
I20260928 12:29:53.807693 26413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384: No bootstrap required, opened a new log
I20260928 12:29:53.809942 26413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fdbbff3d2c47fb961045f7a6d48384" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45111 } }
I20260928 12:29:53.810137 26413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:53.810184 26413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fdbbff3d2c47fb961045f7a6d48384, State: Initialized, Role: FOLLOWER
I20260928 12:29:53.810357 26413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fdbbff3d2c47fb961045f7a6d48384" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45111 } }
I20260928 12:29:53.810441 26413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:53.810473 26413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:53.810523 26413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:53.811285 26413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fdbbff3d2c47fb961045f7a6d48384" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45111 } }
I20260928 12:29:53.811398 26413 leader_election.cc:302] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12fdbbff3d2c47fb961045f7a6d48384; no voters:
I20260928 12:29:53.811553 26413 leader_election.cc:288] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:53.811611 26418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:53.811790 26413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:53.811803 26418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [term 1 LEADER]: Becoming Leader. State: Replica: 12fdbbff3d2c47fb961045f7a6d48384, State: Running, Role: LEADER
I20260928 12:29:53.811944 26418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fdbbff3d2c47fb961045f7a6d48384" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45111 } }
I20260928 12:29:53.812292 26419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12fdbbff3d2c47fb961045f7a6d48384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fdbbff3d2c47fb961045f7a6d48384" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45111 } } }
I20260928 12:29:53.812368 26419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:53.812598 26420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12fdbbff3d2c47fb961045f7a6d48384. Latest consensus state: current_term: 1 leader_uuid: "12fdbbff3d2c47fb961045f7a6d48384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fdbbff3d2c47fb961045f7a6d48384" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45111 } } }
I20260928 12:29:53.812732 26420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384 [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:53.812898 26425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:53.813429 26425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:53.815095 26425 catalog_manager.cc:1347] Generated new cluster ID: 21b5d4aa66e5420db3eef597d62b5c30
I20260928 12:29:53.815155 26425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:29:53 dist-test-slave-cb7x krb5kdc[26332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:53.820034 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:53.807751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.7.19.126:45111)
0928 12:29:53.807937 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:53.807947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:53.807981 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:29:53.809060 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:53.809070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:53.809140 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:53.809294 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:53.809906 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:53.811777 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:53.813165 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:53.814006 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:53.814125 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:53.816611 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:53.816616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:53.816616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:53.816628 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:53.818425 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:53.818810 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:53.818813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:53.818813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:53.818903 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:53.819114 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:53.819115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:53.819115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:53.819351 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:53.819565 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:53.819633 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 12:29:53.819781 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 12:29:53.823239 26425 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:53.823761 26425 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:53.829169 26425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384: Generated new TSK 0
I20260928 12:29:53.829337 26425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.19.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
Entry for principal kudu/127.7.19.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab.
I20260928 12:29:53.868306 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45111
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:53.947826 26439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:53.948058 26439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:53.948094 26439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:53.949290 26439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:53.949354 26439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:53.949398 26439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:53.950843 26439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.65.keytab
--principal=kudu/127.7.19.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:53.951105 26439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:53.951334 26439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:53.956437 26444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:53.956457 26445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:53.956539 26447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:53.956970 26439 server_base.cc:1034] running on GCE node
I20260928 12:29:53.957129 26439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:53.957396 26439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:53.958570 26439 hybrid_clock.cc:648] HybridClock initialized: now 1790598593958515 us; error 63 us; skew 500 ppm
Sep 28 12:29:53 dist-test-slave-cb7x krb5kdc[26332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598593, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:53.961218 26439 init.cc:377] Logged in from keytab as kudu/127.7.19.65@KRBTEST.COM (short username kudu)
I20260928 12:29:53.961685 26439 webserver.cc:460] Webserver started at http://127.7.19.65:41881/ using document root <none> and password file <none>
I20260928 12:29:53.961941 26439 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:53.962018 26439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:53.962175 26439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:53.963369 26439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "63d30bb356fa4c7fb808798373757bff"
format_stamp: "Formatted at 2026-09-28 12:29:53 on dist-test-slave-cb7x"
server_key: "7d18578136188320ee610ce36cf55cbd"
server_key_iv: "b89e6f59fe762e78d42767e3c8552613"
server_key_version: "encryptionkey@0"
I20260928 12:29:53.963757 26439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "63d30bb356fa4c7fb808798373757bff"
format_stamp: "Formatted at 2026-09-28 12:29:53 on dist-test-slave-cb7x"
server_key: "7d18578136188320ee610ce36cf55cbd"
server_key_iv: "b89e6f59fe762e78d42767e3c8552613"
server_key_version: "encryptionkey@0"
I20260928 12:29:53.965269 26439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 12:29:53.966012 26454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:53.966178 26439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:29:53.966305 26439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "63d30bb356fa4c7fb808798373757bff"
format_stamp: "Formatted at 2026-09-28 12:29:53 on dist-test-slave-cb7x"
server_key: "7d18578136188320ee610ce36cf55cbd"
server_key_iv: "b89e6f59fe762e78d42767e3c8552613"
server_key_version: "encryptionkey@0"
I20260928 12:29:53.966384 26439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:53.987498 26439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:53.989778 26439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:53.989952 26439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:53.990192 26439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:53.990514 26439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:53.990572 26439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:53.990608 26439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:53.990638 26439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:53.996706 26439 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:46237
I20260928 12:29:53.996762 26567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:46237 every 8 connection(s)
I20260928 12:29:53.997124 26439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:54.002966 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26439
I20260928 12:29:54.003048 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
Sep 28 12:29:53 dist-test-slave-cb7x krb5kdc[26332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598593, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.65@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:54.003232 7245 external_mini_cluster.cc:1428] Setting key 57327dab1c32a90ac44b26c946df7697
I20260928 12:29:54.006704 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:53.997861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:50879 (local address 127.7.19.126:45111)
0928 12:29:53.997945 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:53.997948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:53.998105 (+ 157us) server_negotiation.cc:407] Connection header received
0928 12:29:53.998629 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:53.998630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:53.998641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:53.998694 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:53.999638 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.000149 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.001230 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.001460 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.001546 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.003698 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.003699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.003700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:54.003707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.005236 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.005692 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.005693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.005693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.005760 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.006010 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.006011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.006012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.006158 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.006234 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.006568 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.006635 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 12:29:54.007023 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:53.997809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45111 (local address 127.7.19.65:50879)
0928 12:29:53.997988 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:53.998023 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:53.998525 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:53.998785 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:53.998788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:53.999346 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:53.999532 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:53.999537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.000243 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.000244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.001127 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.001130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.001557 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.001558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.001675 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.002132 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:54.002140 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:54.003615 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:54.005341 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:54.005349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:54.005351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:54.005619 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:54.005842 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:54.005843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:54.005843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:54.005945 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:54.006229 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:54.006233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:54.006425 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.006544 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.006729 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260928 12:29:54.007573 26568 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45111
I20260928 12:29:54.007666 26568 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:54.007859 26568 heartbeater.cc:511] Master 127.7.19.126:45111 requested a full tablet report, sending...
I20260928 12:29:54.008363 26377 ts_manager.cc:194] Registered new tserver with Master: 63d30bb356fa4c7fb808798373757bff (127.7.19.65:46237)
I20260928 12:29:54.009042 26377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.65@KRBTEST.COM'} at 127.7.19.65:50879
WARNING: no policy specified for kudu/127.7.19.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
Entry for principal kudu/127.7.19.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab.
I20260928 12:29:54.030933 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45111
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:54.109746 26573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:54.109927 26573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:54.109951 26573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:54.110981 26573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:54.111012 26573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:54.111037 26573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:54.112591 26573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.66.keytab
--principal=kudu/127.7.19.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:54.112829 26573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:54.113080 26573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:54.118046 26578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.118053 26579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.118091 26581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.118088 26573 server_base.cc:1034] running on GCE node
I20260928 12:29:54.118422 26573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:54.118660 26573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:54.119834 26573 hybrid_clock.cc:648] HybridClock initialized: now 1790598594119834 us; error 42 us; skew 500 ppm
Sep 28 12:29:54 dist-test-slave-cb7x krb5kdc[26332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:54.122244 26573 init.cc:377] Logged in from keytab as kudu/127.7.19.66@KRBTEST.COM (short username kudu)
I20260928 12:29:54.122660 26573 webserver.cc:460] Webserver started at http://127.7.19.66:41329/ using document root <none> and password file <none>
I20260928 12:29:54.122895 26573 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:54.122963 26573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:54.123113 26573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:54.124132 26573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "4a89039003a54087a77fc01dadb63c72"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "4c303d1f6c97107340ec5b779a19f103"
server_key_iv: "3d964db374e9c7b01a0f37138678013f"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.124495 26573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "4a89039003a54087a77fc01dadb63c72"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "4c303d1f6c97107340ec5b779a19f103"
server_key_iv: "3d964db374e9c7b01a0f37138678013f"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.125859 26573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:54.126552 26588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.126734 26573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:29:54.126834 26573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "4a89039003a54087a77fc01dadb63c72"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "4c303d1f6c97107340ec5b779a19f103"
server_key_iv: "3d964db374e9c7b01a0f37138678013f"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.126922 26573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:54.138979 26573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:54.140805 26573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:54.140962 26573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:54.141196 26573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:54.141505 26573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:54.141557 26573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.141599 26573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:54.141628 26573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.147465 26573 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:41551
I20260928 12:29:54.147564 26701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:41551 every 8 connection(s)
I20260928 12:29:54.147890 26573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
Sep 28 12:29:54 dist-test-slave-cb7x krb5kdc[26332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.66@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:54.155365 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26573
I20260928 12:29:54.155457 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:29:54.155676 7245 external_mini_cluster.cc:1428] Setting key 661a173546bd3a596ac6715db033db29
I20260928 12:29:54.157548 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.148663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:49757 (local address 127.7.19.126:45111)
0928 12:29:54.148722 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.148724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.148952 (+ 228us) server_negotiation.cc:407] Connection header received
0928 12:29:54.149436 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.149438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.149450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.149511 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:54.150410 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.150959 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.151917 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.152116 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.152175 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.154389 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.154391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.154391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:54.154398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.155957 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.156410 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.156411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.156412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.156486 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.156848 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.156849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.156850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.156997 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.157056 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.157392 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.157468 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:29:54.157819 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.148637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45111 (local address 127.7.19.66:49757)
0928 12:29:54.148842 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.148877 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.149344 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.149564 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.149567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.150111 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:54.150321 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.150326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.151045 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.151046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.151833 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.151835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.152193 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.152194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.152304 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.152866 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:54.152877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:54.154305 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:54.156035 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:54.156042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:54.156044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:54.156333 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:54.156557 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:54.156558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:54.156559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:54.156739 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:54.157068 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:54.157075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:54.157242 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.157349 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.157520 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":144,"threads_started":1}
I20260928 12:29:54.158448 26702 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45111
I20260928 12:29:54.158528 26702 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:54.158721 26702 heartbeater.cc:511] Master 127.7.19.126:45111 requested a full tablet report, sending...
I20260928 12:29:54.159135 26377 ts_manager.cc:194] Registered new tserver with Master: 4a89039003a54087a77fc01dadb63c72 (127.7.19.66:41551)
I20260928 12:29:54.159714 26377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.66@KRBTEST.COM'} at 127.7.19.66:49757
WARNING: no policy specified for kudu/127.7.19.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
Entry for principal kudu/127.7.19.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab.
I20260928 12:29:54.181694 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45111
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:54.262565 26707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:54.262825 26707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:54.262864 26707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:54.264202 26707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:54.264273 26707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:54.264318 26707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:54.265854 26707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.67.keytab
--principal=kudu/127.7.19.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:54.266140 26707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:54.266381 26707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:54.271739 26715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.271741 26712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.271770 26713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.272357 26707 server_base.cc:1034] running on GCE node
I20260928 12:29:54.272562 26707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:54.272879 26707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:54.274052 26707 hybrid_clock.cc:648] HybridClock initialized: now 1790598594274047 us; error 40 us; skew 500 ppm
Sep 28 12:29:54 dist-test-slave-cb7x krb5kdc[26332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:54.276821 26707 init.cc:377] Logged in from keytab as kudu/127.7.19.67@KRBTEST.COM (short username kudu)
I20260928 12:29:54.277351 26707 webserver.cc:460] Webserver started at http://127.7.19.67:44903/ using document root <none> and password file <none>
I20260928 12:29:54.277621 26707 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:54.277702 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:54.277841 26707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:54.278932 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "e01e2206db2a4ccba7ae281e227072b6"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "7c7c3b97d48aa9be7136119fb9cac91b"
server_key_iv: "cf65223e5e3e99e71717186d30abdd30"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.279346 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "e01e2206db2a4ccba7ae281e227072b6"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "7c7c3b97d48aa9be7136119fb9cac91b"
server_key_iv: "cf65223e5e3e99e71717186d30abdd30"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.280750 26707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:54.281551 26722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.281734 26707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:54.281831 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "e01e2206db2a4ccba7ae281e227072b6"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "7c7c3b97d48aa9be7136119fb9cac91b"
server_key_iv: "cf65223e5e3e99e71717186d30abdd30"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.281910 26707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:54.291146 26707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:54.293290 26707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:54.293445 26707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:54.293668 26707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:54.294023 26707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:54.294067 26707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.294100 26707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:54.294123 26707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.300187 26707 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:40041
I20260928 12:29:54.300223 26835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:40041 every 8 connection(s)
I20260928 12:29:54.300540 26707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
Sep 28 12:29:54 dist-test-slave-cb7x krb5kdc[26332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598594, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.67@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:29:54.306329 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26707
I20260928 12:29:54.306428 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:29:54.306628 7245 external_mini_cluster.cc:1428] Setting key 565611bdfea083945b1c3bb593e0e331
I20260928 12:29:54.309610 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.301305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:33203 (local address 127.7.19.126:45111)
0928 12:29:54.301388 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.301390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.301547 (+ 157us) server_negotiation.cc:407] Connection header received
0928 12:29:54.301993 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.301995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.302005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.302051 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:54.302954 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.303440 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.304422 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.304595 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.304650 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.306877 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.306879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.306879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:54.306887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.308331 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.308840 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.308841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.308841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.308891 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.309096 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.309096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.309096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.309195 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.309245 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.309490 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.309548 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 12:29:54.309839 26838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.301266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:45111 (local address 127.7.19.67:33203)
0928 12:29:54.301436 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.301469 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.301918 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.302117 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.302120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.302699 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:54.302882 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.302886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.303522 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.303523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.304348 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.304350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.304663 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.304664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.304819 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.305253 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:29:54.305261 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:54.306771 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:54.308429 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:54.308435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:54.308436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:54.308778 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:54.308935 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:29:54.308935 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:29:54.308935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:29:54.309041 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:29:54.309253 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:54.309257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:54.309371 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.309454 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.309571 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260928 12:29:54.310329 26836 heartbeater.cc:347] Connected to a master server at 127.7.19.126:45111
I20260928 12:29:54.310410 26836 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:54.310603 26836 heartbeater.cc:511] Master 127.7.19.126:45111 requested a full tablet report, sending...
I20260928 12:29:54.310989 26377 ts_manager.cc:194] Registered new tserver with Master: e01e2206db2a4ccba7ae281e227072b6 (127.7.19.67:40041)
I20260928 12:29:54.311331 26377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.19.67@KRBTEST.COM'} at 127.7.19.67:33203
I20260928 12:29:54.318423 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:54.325476 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.319793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.7.19.126:45111)
0928 12:29:54.319864 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.319866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.320088 (+ 222us) server_negotiation.cc:407] Connection header received
0928 12:29:54.320141 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.320142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.320151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.320189 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:54.320741 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.321212 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.321926 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.322096 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.322144 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.323325 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.323326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.323326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:54.323331 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.324473 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.324846 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.324846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.324847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.324895 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:54.325043 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:54.325043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:54.325044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:54.325180 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.325264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.325336 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.325390 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:29:54.327126 26377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:54.328644 26377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:54.335196 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.330971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:46237 (local address 127.0.0.1:44868)
0928 12:29:54.331271 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.331315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.331366 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.331633 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.331636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.331638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:54.331718 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.331720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.333075 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.333076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.334172 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.334174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.334902 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.334903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.334979 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.334992 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.335065 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.335119 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"thread_start_us":44,"threads_started":1}
I20260928 12:29:54.335209 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.331078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.7.19.65:46237)
0928 12:29:54.331483 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.331488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.331497 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:54.331538 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.331540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.331552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.331612 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:54.331789 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.332965 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.334277 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.334814 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.334904 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.334967 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.335086 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.335129 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":93,"threads_started":1}
I20260928 12:29:54.335810 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.331010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:41551 (local address 127.0.0.1:36262)
0928 12:29:54.331271 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.331313 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.331371 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.331934 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.331934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.331936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:54.331986 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.331988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.333218 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.333219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.334306 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.334309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.335570 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.335571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.335630 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.335634 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.335698 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.335738 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":46,"threads_started":1}
I20260928 12:29:54.335850 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.331105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.7.19.66:41551)
0928 12:29:54.331712 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.331715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.331760 (+ 45us) server_negotiation.cc:407] Connection header received
0928 12:29:54.331799 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.331801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.331817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.331889 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:54.332055 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.333109 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.334387 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.335088 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.335155 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.335215 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.335726 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.335775 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":81,"threads_started":1}
I20260928 12:29:54.335932 26502 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3816878e184611a428de048f53085d (DEFAULT_TABLE table=test-table [id=a9fe3daa3b22410d801f87e2b158129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:54.336350 26502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3816878e184611a428de048f53085d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:54.336926 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.331171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40041 (local address 127.0.0.1:53726)
0928 12:29:54.331983 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.331993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.332043 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.332886 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.332887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.332889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:54.332926 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.332927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.334127 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.334128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.335226 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.335229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.336650 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.336651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.336763 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.336767 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.336819 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.336864 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":44,"threads_started":1}
I20260928 12:29:54.337153 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.331286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.7.19.67:40041)
0928 12:29:54.331467 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.331472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.332727 (+ 1255us) server_negotiation.cc:407] Connection header received
0928 12:29:54.332780 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.332781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.332794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.332873 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:54.332978 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.334026 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.335457 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.336517 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.336575 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.336610 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.337022 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.337070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260928 12:29:54.337505 26636 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3816878e184611a428de048f53085d (DEFAULT_TABLE table=test-table [id=a9fe3daa3b22410d801f87e2b158129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:54.337759 26636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3816878e184611a428de048f53085d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:54.339303 26853 tablet_bootstrap.cc:492] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff: Bootstrap starting.
I20260928 12:29:54.339866 26770 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3816878e184611a428de048f53085d (DEFAULT_TABLE table=test-table [id=a9fe3daa3b22410d801f87e2b158129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:54.340000 26853 tablet_bootstrap.cc:654] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:54.340077 26770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3816878e184611a428de048f53085d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:54.340584 26853 log.cc:824] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:54.340745 26854 tablet_bootstrap.cc:492] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Bootstrap starting.
I20260928 12:29:54.341298 26853 tablet_bootstrap.cc:492] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff: No bootstrap required, opened a new log
I20260928 12:29:54.341387 26853 ts_tablet_manager.cc:1397] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:54.341393 26854 tablet_bootstrap.cc:654] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:54.341944 26854 log.cc:824] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:54.342700 26854 tablet_bootstrap.cc:492] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: No bootstrap required, opened a new log
I20260928 12:29:54.342784 26854 ts_tablet_manager.cc:1397] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:54.343174 26857 tablet_bootstrap.cc:492] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Bootstrap starting.
I20260928 12:29:54.343091 26853 raft_consensus.cc:348] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.343264 26853 raft_consensus.cc:374] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:54.343307 26853 raft_consensus.cc:729] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d30bb356fa4c7fb808798373757bff, State: Initialized, Role: FOLLOWER
I20260928 12:29:54.343415 26853 consensus_queue.cc:260] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.343713 26853 ts_tablet_manager.cc:1428] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 12:29:54.343868 26568 heartbeater.cc:503] Master 127.7.19.126:45111 was elected leader, sending a full tablet report...
I20260928 12:29:54.344012 26857 tablet_bootstrap.cc:654] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:54.344570 26857 log.cc:824] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:54.344863 26854 raft_consensus.cc:348] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.345052 26854 raft_consensus.cc:374] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:54.345104 26854 raft_consensus.cc:729] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a89039003a54087a77fc01dadb63c72, State: Initialized, Role: FOLLOWER
I20260928 12:29:54.345212 26854 consensus_queue.cc:260] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.345275 26857 tablet_bootstrap.cc:492] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: No bootstrap required, opened a new log
I20260928 12:29:54.345352 26857 ts_tablet_manager.cc:1397] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:54.345496 26854 ts_tablet_manager.cc:1428] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 12:29:54.345508 26702 heartbeater.cc:503] Master 127.7.19.126:45111 was elected leader, sending a full tablet report...
I20260928 12:29:54.346890 26857 raft_consensus.cc:348] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.347011 26857 raft_consensus.cc:374] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:54.347031 26857 raft_consensus.cc:729] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01e2206db2a4ccba7ae281e227072b6, State: Initialized, Role: FOLLOWER
I20260928 12:29:54.347159 26857 consensus_queue.cc:260] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.347412 26857 ts_tablet_manager.cc:1428] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:54.347488 26836 heartbeater.cc:503] Master 127.7.19.126:45111 was elected leader, sending a full tablet report...
W20260928 12:29:54.398499 26703 tablet.cc:2369] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:29:54.440598 26858 raft_consensus.cc:482] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:54.440805 26858 raft_consensus.cc:504] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.441221 26858 leader_election.cc:288] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a89039003a54087a77fc01dadb63c72 (127.7.19.66:41551), e01e2206db2a4ccba7ae281e227072b6 (127.7.19.67:40041)
I20260928 12:29:54.444976 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.441480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:58145 (local address 127.7.19.66:41551)
0928 12:29:54.441539 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.441541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.441554 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:29:54.441588 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.441589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.441601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.441670 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:54.441855 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.442629 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.443997 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.444587 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.444660 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.444693 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.444862 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.444904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 12:29:54.444976 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.441509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:57087 (local address 127.7.19.67:40041)
0928 12:29:54.441557 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.441559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.441833 (+ 274us) server_negotiation.cc:407] Connection header received
0928 12:29:54.441863 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.441864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.441877 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.441934 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:54.442057 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.442754 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.443995 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.444582 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.444660 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.444692 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.444861 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.444905 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 12:29:54.444993 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.441399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:41551 (local address 127.7.19.65:58145)
0928 12:29:54.441468 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.441483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.441532 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.441692 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.441693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.441694 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:54.441771 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.441774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.442708 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.442709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.443899 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.443902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.444692 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.444693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.444775 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.444782 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.444840 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.444886 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":9}
I20260928 12:29:54.444999 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.441399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:40041 (local address 127.7.19.65:57087)
0928 12:29:54.441712 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.441721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.441804 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.441943 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.441944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.441945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:54.441992 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.441993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.442822 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.442823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.443900 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.443902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.444692 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.444693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.444775 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.444782 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.444840 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.444885 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":103,"thread_start_us":119,"threads_started":1}
I20260928 12:29:54.445415 26656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3816878e184611a428de048f53085d" candidate_uuid: "63d30bb356fa4c7fb808798373757bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a89039003a54087a77fc01dadb63c72" is_pre_election: true
I20260928 12:29:54.445604 26656 raft_consensus.cc:2454] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d30bb356fa4c7fb808798373757bff in term 0.
I20260928 12:29:54.445595 26790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3816878e184611a428de048f53085d" candidate_uuid: "63d30bb356fa4c7fb808798373757bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01e2206db2a4ccba7ae281e227072b6" is_pre_election: true
I20260928 12:29:54.445730 26790 raft_consensus.cc:2454] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d30bb356fa4c7fb808798373757bff in term 0.
I20260928 12:29:54.445919 26455 leader_election.cc:302] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a89039003a54087a77fc01dadb63c72, 63d30bb356fa4c7fb808798373757bff; no voters:
I20260928 12:29:54.446055 26858 raft_consensus.cc:2781] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:54.446110 26858 raft_consensus.cc:482] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:54.446137 26858 raft_consensus.cc:3037] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:54.446892 26858 raft_consensus.cc:504] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.447038 26858 leader_election.cc:288] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [CANDIDATE]: Term 1 election: Requested vote from peers 4a89039003a54087a77fc01dadb63c72 (127.7.19.66:41551), e01e2206db2a4ccba7ae281e227072b6 (127.7.19.67:40041)
I20260928 12:29:54.447196 26790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3816878e184611a428de048f53085d" candidate_uuid: "63d30bb356fa4c7fb808798373757bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01e2206db2a4ccba7ae281e227072b6"
I20260928 12:29:54.447211 26656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3816878e184611a428de048f53085d" candidate_uuid: "63d30bb356fa4c7fb808798373757bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a89039003a54087a77fc01dadb63c72"
I20260928 12:29:54.447291 26790 raft_consensus.cc:3037] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:54.447295 26656 raft_consensus.cc:3037] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:54.447960 26790 raft_consensus.cc:2454] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d30bb356fa4c7fb808798373757bff in term 1.
I20260928 12:29:54.448124 26656 raft_consensus.cc:2454] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d30bb356fa4c7fb808798373757bff in term 1.
I20260928 12:29:54.448201 26458 leader_election.cc:302] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d30bb356fa4c7fb808798373757bff, e01e2206db2a4ccba7ae281e227072b6; no voters:
I20260928 12:29:54.448302 26858 raft_consensus.cc:2781] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:54.448439 26858 raft_consensus.cc:686] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [term 1 LEADER]: Becoming Leader. State: Replica: 63d30bb356fa4c7fb808798373757bff, State: Running, Role: LEADER
I20260928 12:29:54.448537 26858 consensus_queue.cc:237] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } }
I20260928 12:29:54.449406 26377 catalog_manager.cc:5495] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d30bb356fa4c7fb808798373757bff (127.7.19.65). New cstate: current_term: 1 leader_uuid: "63d30bb356fa4c7fb808798373757bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63d30bb356fa4c7fb808798373757bff" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 } health_report { overall_health: UNKNOWN } } }
I20260928 12:29:54.479513 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.477037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.7.19.65:46237)
0928 12:29:54.477104 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.477106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.477115 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:54.477185 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.477186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.477207 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.477285 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:54.477420 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.477964 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.478735 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.478928 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.479000 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.479174 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:54.479271 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:54.479365 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.479384 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.479404 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:29:54.481385 26790 raft_consensus.cc:1261] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 1 FOLLOWER]: Refusing update from remote peer 63d30bb356fa4c7fb808798373757bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:54.481385 26656 raft_consensus.cc:1261] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 1 FOLLOWER]: Refusing update from remote peer 63d30bb356fa4c7fb808798373757bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:54.481762 26863 consensus_queue.cc:1035] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a89039003a54087a77fc01dadb63c72" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 41551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:54.481925 26858 consensus_queue.cc:1035] T 9f3816878e184611a428de048f53085d P 63d30bb356fa4c7fb808798373757bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01e2206db2a4ccba7ae281e227072b6" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:54.484036 26866 mvcc.cc:205] Tried to move back new op lower bound from 7334291842992222208 to 7334291842861682688. Current Snapshot: MvccSnapshot[applied={T|T < 7334291842992222208}]
I20260928 12:29:54.484651 26868 mvcc.cc:205] Tried to move back new op lower bound from 7334291842992222208 to 7334291842861682688. Current Snapshot: MvccSnapshot[applied={T|T < 7334291842992222208}]
I20260928 12:29:54.486886 26377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:54.486984 26377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:54.487677 26377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12fdbbff3d2c47fb961045f7a6d48384: Sending DeleteTablet for 3 replicas of tablet 9f3816878e184611a428de048f53085d
I20260928 12:29:54.488005 26636 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3816878e184611a428de048f53085d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:54 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:36262
I20260928 12:29:54.488029 26770 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3816878e184611a428de048f53085d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:54 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:53726
I20260928 12:29:54.488255 26877 tablet_replica.cc:330] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: stopping tablet replica
I20260928 12:29:54.488281 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26439
I20260928 12:29:54.488298 26502 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3816878e184611a428de048f53085d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:54 UTC) from {username='kudu', principal='kudu/127.7.19.126@KRBTEST.COM'} at 127.0.0.1:44868
I20260928 12:29:54.488356 26877 raft_consensus.cc:2229] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:54.488446 26877 raft_consensus.cc:2258] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:54.488569 26876 tablet_replica.cc:330] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: stopping tablet replica
I20260928 12:29:54.488638 26876 raft_consensus.cc:2229] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:54.488838 26876 raft_consensus.cc:2258] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:54.489486 26876 ts_tablet_manager.cc:1905] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:54.490218 26877 ts_tablet_manager.cc:1905] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:54.491074 26876 ts_tablet_manager.cc:1918] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:54.491150 26876 log.cc:1196] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/9f3816878e184611a428de048f53085d
I20260928 12:29:54.491429 26876 ts_tablet_manager.cc:1939] T 9f3816878e184611a428de048f53085d P e01e2206db2a4ccba7ae281e227072b6: Deleting consensus metadata
I20260928 12:29:54.491833 26877 ts_tablet_manager.cc:1918] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:54.491825 26364 catalog_manager.cc:4841] TS e01e2206db2a4ccba7ae281e227072b6 (127.7.19.67:40041): tablet 9f3816878e184611a428de048f53085d (table test-table [id=a9fe3daa3b22410d801f87e2b158129c]) successfully deleted
I20260928 12:29:54.491906 26877 log.cc:1196] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/9f3816878e184611a428de048f53085d
I20260928 12:29:54.492123 26877 ts_tablet_manager.cc:1939] T 9f3816878e184611a428de048f53085d P 4a89039003a54087a77fc01dadb63c72: Deleting consensus metadata
I20260928 12:29:54.492455 26361 catalog_manager.cc:4841] TS 4a89039003a54087a77fc01dadb63c72 (127.7.19.66:41551): tablet 9f3816878e184611a428de048f53085d (table test-table [id=a9fe3daa3b22410d801f87e2b158129c]) successfully deleted
W20260928 12:29:54.494529 26364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:46237 (error 108)
I20260928 12:29:54.494748 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26573
I20260928 12:29:54.499486 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26707
I20260928 12:29:54.499619 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.499457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:46237 (local address 127.0.0.1:44892)
0928 12:29:54.499533 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.499577 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 12:29:54.500936 26364 catalog_manager.cc:4584] TS 63d30bb356fa4c7fb808798373757bff (127.7.19.65:46237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f3816878e184611a428de048f53085d: Network error: Client connection negotiation failed: client connection to 127.7.19.65:46237: connect: Connection refused (error 111)
I20260928 12:29:54.504124 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26345
2026-09-28T12:29:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1986 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T12:29:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:54Z Disabled control of system clock
I20260928 12:29:54.530607 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41569
--webserver_interface=127.7.19.126
--webserver_port=0
--builtin_ntp_servers=127.7.19.84:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:41569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:54.610376 26882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:54.610586 26882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:54.610608 26882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:54.611672 26882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:54.611706 26882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:54.611719 26882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:54.611732 26882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:54.611747 26882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:54.613021 26882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:41569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:41569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:54.613209 26882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:54.613449 26882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:54.618315 26890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.618352 26887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.618413 26882 server_base.cc:1034] running on GCE node
W20260928 12:29:54.618372 26888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.618693 26882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:54.618981 26882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:54.620187 26882 hybrid_clock.cc:648] HybridClock initialized: now 1790598594620184 us; error 38 us; skew 500 ppm
I20260928 12:29:54.621402 26882 webserver.cc:460] Webserver started at http://127.7.19.126:45087/ using document root <none> and password file <none>
I20260928 12:29:54.621639 26882 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:54.621711 26882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:54.621881 26882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:54.622911 26882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "280baeabda4f4cbf82ca05deb8a3307c"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "c8cfd625177855d9686b55c28ee88c0d"
server_key_iv: "cdb6680afd1f87140e8a530cdde2e138"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.623265 26882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "280baeabda4f4cbf82ca05deb8a3307c"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "c8cfd625177855d9686b55c28ee88c0d"
server_key_iv: "cdb6680afd1f87140e8a530cdde2e138"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.624485 26882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:29:54.625295 26896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.625473 26882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.625564 26882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "280baeabda4f4cbf82ca05deb8a3307c"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "c8cfd625177855d9686b55c28ee88c0d"
server_key_iv: "cdb6680afd1f87140e8a530cdde2e138"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.625649 26882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:54.633939 26882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:54.634210 26882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:54.634379 26882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:54.638280 26882 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:41569
I20260928 12:29:54.638319 26948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:41569 every 8 connection(s)
I20260928 12:29:54.638648 26882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:54.639539 26949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:54.641763 26949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c: Bootstrap starting.
I20260928 12:29:54.642270 26949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:54.642508 26949 log.cc:824] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:54.643101 26949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c: No bootstrap required, opened a new log
I20260928 12:29:54.644732 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26882
I20260928 12:29:54.644606 26949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280baeabda4f4cbf82ca05deb8a3307c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41569 } }
I20260928 12:29:54.644822 26949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:54.644852 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:54.644891 26949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280baeabda4f4cbf82ca05deb8a3307c, State: Initialized, Role: FOLLOWER
I20260928 12:29:54.645047 26949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280baeabda4f4cbf82ca05deb8a3307c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41569 } }
I20260928 12:29:54.645090 7245 external_mini_cluster.cc:1428] Setting key e2e5fc0f3d527ff342417fe8a4c2a627
I20260928 12:29:54.645108 26949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:54.645151 26949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:54.645200 26949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:54.645907 26949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280baeabda4f4cbf82ca05deb8a3307c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41569 } }
I20260928 12:29:54.646011 26949 leader_election.cc:302] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280baeabda4f4cbf82ca05deb8a3307c; no voters:
I20260928 12:29:54.646163 26949 leader_election.cc:288] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:54.646273 26954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:54.646417 26949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:54.646497 26954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [term 1 LEADER]: Becoming Leader. State: Replica: 280baeabda4f4cbf82ca05deb8a3307c, State: Running, Role: LEADER
I20260928 12:29:54.646636 26954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280baeabda4f4cbf82ca05deb8a3307c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41569 } }
I20260928 12:29:54.647154 26955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280baeabda4f4cbf82ca05deb8a3307c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280baeabda4f4cbf82ca05deb8a3307c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41569 } } }
I20260928 12:29:54.647226 26955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:54.647558 26956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280baeabda4f4cbf82ca05deb8a3307c. Latest consensus state: current_term: 1 leader_uuid: "280baeabda4f4cbf82ca05deb8a3307c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280baeabda4f4cbf82ca05deb8a3307c" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 41569 } } }
I20260928 12:29:54.647609 26956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:54.647909 26964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:54.648406 26964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:54.649997 26964 catalog_manager.cc:1347] Generated new cluster ID: 6400112714b64b2a8b984b437e0d4a96
I20260928 12:29:54.650048 26964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:54.651397 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.645592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.7.19.126:41569)
0928 12:29:54.645848 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.645856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.645890 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:29:54.646004 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.646011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.646080 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.646237 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:54.646488 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.648196 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.649297 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.649909 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.650032 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.650697 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.650701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.650701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:54.650909 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.650950 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:54.650963 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.651026 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.651124 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.651196 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":120,"threads_started":1}
I20260928 12:29:54.661080 26964 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:54.661764 26964 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:54.669222 26964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c: Generated new TSK 0
I20260928 12:29:54.669386 26964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:29:54.670914 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.65:0
--local_ip_for_outbound_sockets=127.7.19.65
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41569
--builtin_ntp_servers=127.7.19.84:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:54.754559 26973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:54.754854 26973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:54.754894 26973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:54.756039 26973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:54.756103 26973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:54.756147 26973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.65
I20260928 12:29:54.757604 26973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.19.65
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.65
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:54.757864 26973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:54.758168 26973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:54.763942 26979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.764122 26981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.763980 26978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.764219 26973 server_base.cc:1034] running on GCE node
I20260928 12:29:54.764458 26973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:54.764737 26973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:54.765906 26973 hybrid_clock.cc:648] HybridClock initialized: now 1790598594765895 us; error 28 us; skew 500 ppm
I20260928 12:29:54.767316 26973 webserver.cc:460] Webserver started at http://127.7.19.65:40331/ using document root <none> and password file <none>
I20260928 12:29:54.767589 26973 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:54.767647 26973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:54.767791 26973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:54.769157 26973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data/instance:
uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "577dcb4f63f3211fe2c3b8f10fe02a2d"
server_key_iv: "c74ee7db38637d9b1fe25cc1ce91c937"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.769529 26973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance:
uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "577dcb4f63f3211fe2c3b8f10fe02a2d"
server_key_iv: "c74ee7db38637d9b1fe25cc1ce91c937"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.771216 26973 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 12:29:54.772051 26987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.772243 26973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:54.772351 26973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal
uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "577dcb4f63f3211fe2c3b8f10fe02a2d"
server_key_iv: "c74ee7db38637d9b1fe25cc1ce91c937"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.772428 26973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:54.794461 26973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:54.794792 26973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:54.794946 26973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:54.795185 26973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:54.795531 26973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:54.795563 26973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.795616 26973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:54.795642 26973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.801331 26973 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.65:36013
I20260928 12:29:54.801409 27100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.65:36013 every 8 connection(s)
I20260928 12:29:54.801749 26973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/data/info.pb
I20260928 12:29:54.805953 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 26973
I20260928 12:29:54.806075 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-0/wal/instance
I20260928 12:29:54.806257 7245 external_mini_cluster.cc:1428] Setting key 7d57e16549d90b35c8e992db25ca0007
I20260928 12:29:54.806969 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.802509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:42229 (local address 127.7.19.126:41569)
0928 12:29:54.802573 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.802575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.802837 (+ 262us) server_negotiation.cc:407] Connection header received
0928 12:29:54.803311 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.803312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.803326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.803388 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:54.804024 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.804512 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.805457 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.805642 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.805697 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.806690 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.806692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.806692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:54.806697 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.806714 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:54.806718 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.806809 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.806853 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.806882 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 12:29:54.807192 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.802499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41569 (local address 127.7.19.65:42229)
0928 12:29:54.802704 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.802746 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.803225 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.803444 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.803447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.803455 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:54.803951 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.803955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.804595 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.804596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.805386 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.805387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.805719 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.805719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.805891 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.806517 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:54.806517 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:54.806573 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:54.806599 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:54.806802 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:54.806805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:54.806807 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.806855 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.806950 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260928 12:29:54.807417 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.66:0
--local_ip_for_outbound_sockets=127.7.19.66
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41569
--builtin_ntp_servers=127.7.19.84:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 12:29:54.808873 27101 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41569
I20260928 12:29:54.808962 27101 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:54.809154 27101 heartbeater.cc:511] Master 127.7.19.126:41569 requested a full tablet report, sending...
I20260928 12:29:54.809682 26913 ts_manager.cc:194] Registered new tserver with Master: fefb1fcdb95b49c7bfa8ac3f8a2a5f36 (127.7.19.65:36013)
I20260928 12:29:54.810310 26913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.65:42229
W20260928 12:29:54.887384 27104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:54.887585 27104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:54.887607 27104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:54.888805 27104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:54.888844 27104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:54.888868 27104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.66
I20260928 12:29:54.890182 27104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.19.66
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.66
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:54.890412 27104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:54.890692 27104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:54.895768 27112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:54.895843 27110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.895848 27104 server_base.cc:1034] running on GCE node
W20260928 12:29:54.895785 27109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:54.896193 27104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:54.896401 27104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:54.897547 27104 hybrid_clock.cc:648] HybridClock initialized: now 1790598594897535 us; error 27 us; skew 500 ppm
I20260928 12:29:54.898875 27104 webserver.cc:460] Webserver started at http://127.7.19.66:38803/ using document root <none> and password file <none>
I20260928 12:29:54.899147 27104 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:54.899209 27104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:54.899412 27104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:54.900501 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data/instance:
uuid: "f16a57ee5a174afdb241505b2410df43"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "9d9352c65fe8b82a51605cb860285f82"
server_key_iv: "7a97f3602c3433c645c8f82185fd2ce0"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.900864 27104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance:
uuid: "f16a57ee5a174afdb241505b2410df43"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "9d9352c65fe8b82a51605cb860285f82"
server_key_iv: "7a97f3602c3433c645c8f82185fd2ce0"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.902143 27104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:29:54.902832 27118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.903014 27104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:29:54.903084 27104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal
uuid: "f16a57ee5a174afdb241505b2410df43"
format_stamp: "Formatted at 2026-09-28 12:29:54 on dist-test-slave-cb7x"
server_key: "9d9352c65fe8b82a51605cb860285f82"
server_key_iv: "7a97f3602c3433c645c8f82185fd2ce0"
server_key_version: "encryptionkey@0"
I20260928 12:29:54.903174 27104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:54.916146 27104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:54.916457 27104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:54.916612 27104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:54.916884 27104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:54.917253 27104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:54.917294 27104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.917316 27104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:54.917367 27104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:54.922992 27104 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.66:37169
I20260928 12:29:54.923070 27231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.66:37169 every 8 connection(s)
I20260928 12:29:54.923401 27104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/data/info.pb
I20260928 12:29:54.928431 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.924090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.66:42803 (local address 127.7.19.126:41569)
0928 12:29:54.924180 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:54.924183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:54.924435 (+ 252us) server_negotiation.cc:407] Connection header received
0928 12:29:54.925036 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:54.925037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:54.925047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:54.925101 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:54.925716 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.926191 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.927104 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.927286 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.927336 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.928121 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:54.928122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:54.928122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:54.928126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:54.928138 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:54.928142 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:54.928186 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:54.928297 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 12:29:54.928363 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 12:29:54.928635 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:54.924063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41569 (local address 127.7.19.66:42803)
0928 12:29:54.924279 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:54.924332 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:54.924943 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:54.925155 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:54.925158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:54.925166 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:54.925647 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.925652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.926275 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.926276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.927033 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:54.927035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:54.927357 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:54.927358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:54.927460 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:54.927998 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:54.927998 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:54.928024 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:54.928044 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:54.928213 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:54.928217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:54.928220 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:54.928293 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 12:29:54.928402 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260928 12:29:54.929122 27232 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41569
I20260928 12:29:54.929196 27232 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:54.929392 27232 heartbeater.cc:511] Master 127.7.19.126:41569 requested a full tablet report, sending...
I20260928 12:29:54.929766 26913 ts_manager.cc:194] Registered new tserver with Master: f16a57ee5a174afdb241505b2410df43 (127.7.19.66:37169)
I20260928 12:29:54.930115 26913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.66:42803
I20260928 12:29:54.932199 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27104
I20260928 12:29:54.932308 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal/instance
I20260928 12:29:54.932495 7245 external_mini_cluster.cc:1428] Setting key b7b978ec75c292007b4a76924a0275a8
I20260928 12:29:54.933377 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.19.67:0
--local_ip_for_outbound_sockets=127.7.19.67
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41569
--builtin_ntp_servers=127.7.19.84:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 12:29:55.013757 27235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:55.013967 27235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:55.013988 27235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:55.015130 27235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:55.015168 27235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:55.015194 27235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.19.67
I20260928 12:29:55.016530 27235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.19.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.19.67
--webserver_port=0
--tserver_master_addrs=127.7.19.126:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.19.67
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:55.016811 27235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:55.017105 27235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:55.021992 27241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:55.022020 27243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:55.021952 27240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:55.022143 27235 server_base.cc:1034] running on GCE node
I20260928 12:29:55.022396 27235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:55.022596 27235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:55.023734 27235 hybrid_clock.cc:648] HybridClock initialized: now 1790598595023721 us; error 28 us; skew 500 ppm
I20260928 12:29:55.025055 27235 webserver.cc:460] Webserver started at http://127.7.19.67:39297/ using document root <none> and password file <none>
I20260928 12:29:55.025305 27235 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:55.025354 27235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:55.025509 27235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:55.026566 27235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data/instance:
uuid: "683336067a4b4558aba63e7578d22600"
format_stamp: "Formatted at 2026-09-28 12:29:55 on dist-test-slave-cb7x"
server_key: "7d293eb78b36140e6dfa000f61d9c13f"
server_key_iv: "b8d7d0159ed15ec8a99626e956c9eeb0"
server_key_version: "encryptionkey@0"
I20260928 12:29:55.026918 27235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance:
uuid: "683336067a4b4558aba63e7578d22600"
format_stamp: "Formatted at 2026-09-28 12:29:55 on dist-test-slave-cb7x"
server_key: "7d293eb78b36140e6dfa000f61d9c13f"
server_key_iv: "b8d7d0159ed15ec8a99626e956c9eeb0"
server_key_version: "encryptionkey@0"
I20260928 12:29:55.028231 27235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:29:55.028981 27249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:55.029171 27235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:55.029270 27235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal
uuid: "683336067a4b4558aba63e7578d22600"
format_stamp: "Formatted at 2026-09-28 12:29:55 on dist-test-slave-cb7x"
server_key: "7d293eb78b36140e6dfa000f61d9c13f"
server_key_iv: "b8d7d0159ed15ec8a99626e956c9eeb0"
server_key_version: "encryptionkey@0"
I20260928 12:29:55.029356 27235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:55.038316 27235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:55.038607 27235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:55.038753 27235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:55.038980 27235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 12:29:55.039302 27235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 12:29:55.039338 27235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:55.039359 27235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 12:29:55.039407 27235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:55.045032 27235 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.67:35715
I20260928 12:29:55.045073 27362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.67:35715 every 8 connection(s)
I20260928 12:29:55.045388 27235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/data/info.pb
I20260928 12:29:55.047561 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27235
I20260928 12:29:55.047662 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal/instance
I20260928 12:29:55.047880 7245 external_mini_cluster.cc:1428] Setting key 5703149da11c3e2447d02a254bf3eb15
I20260928 12:29:55.050758 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.046116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.67:33191 (local address 127.7.19.126:41569)
0928 12:29:55.046176 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.046178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.046401 (+ 223us) server_negotiation.cc:407] Connection header received
0928 12:29:55.046927 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.046928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.046939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.046989 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:55.047797 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.048346 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.049502 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.049717 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.049783 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.050514 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:55.050516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:55.050516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:55.050520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:55.050533 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:55.050537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:55.050587 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.050673 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.050701 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:29:55.050904 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.046109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:41569 (local address 127.7.19.67:33191)
0928 12:29:55.046284 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.046316 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:55.046774 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:55.047049 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:55.047053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:55.047061 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:29:55.047695 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.047702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.048416 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.048417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.049404 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.049407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.049802 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.049803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.049908 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.050377 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 12:29:55.050377 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:29:55.050404 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 12:29:55.050425 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:29:55.050599 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:29:55.050601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:29:55.050603 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 12:29:55.050651 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 12:29:55.050722 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260928 12:29:55.051371 27363 heartbeater.cc:347] Connected to a master server at 127.7.19.126:41569
I20260928 12:29:55.051470 27363 heartbeater.cc:464] Registering TS with master...
I20260928 12:29:55.051636 27363 heartbeater.cc:511] Master 127.7.19.126:41569 requested a full tablet report, sending...
I20260928 12:29:55.052016 26913 ts_manager.cc:194] Registered new tserver with Master: 683336067a4b4558aba63e7578d22600 (127.7.19.67:35715)
I20260928 12:29:55.052332 26913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.19.67:33191
I20260928 12:29:55.059710 7245 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 12:29:55.063742 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.061092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.7.19.126:41569)
0928 12:29:55.061156 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.061158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.061229 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:29:55.061283 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.061284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.061293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.061341 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:55.061485 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.061961 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.062680 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.062949 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.063001 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.063562 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:55.063563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:55.063564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:55.063567 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:55.063580 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:55.063583 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:55.063629 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.063664 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.063684 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:29:55.065232 26913 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 12:29:55.066802 26913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 12:29:55.073060 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.069226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.7.19.65:36013)
0928 12:29:55.069424 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.069429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.069525 (+ 96us) server_negotiation.cc:407] Connection header received
0928 12:29:55.069664 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.069667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.069683 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.069773 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:55.069879 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.070672 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.071924 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.072634 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.072762 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.072857 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.072939 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.072976 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260928 12:29:55.073050 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.069004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:36013 (local address 127.0.0.1:37218)
0928 12:29:55.069423 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.069435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:55.069504 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:55.069769 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:55.069770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:55.069772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:55.069816 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.069817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.070750 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.070751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.071843 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.071845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.072762 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.072763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.072836 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.072849 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 12:29:55.072925 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:29:55.072976 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":37,"thread_start_us":92,"threads_started":1}
I20260928 12:29:55.073412 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.068967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:35715 (local address 127.0.0.1:43420)
0928 12:29:55.069171 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.069211 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:55.069287 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:55.070095 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:55.070096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:55.070098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:55.070140 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.070141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.071103 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.071104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.072205 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.072207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.073173 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.073175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.073252 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.073258 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 12:29:55.073303 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 12:29:55.073347 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260928 12:29:55.073777 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.069210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.7.19.67:35715)
0928 12:29:55.069961 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.069964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.069971 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:29:55.070000 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.070001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.070013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.070070 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:55.070215 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.071013 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.072297 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.073054 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.073522 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.073616 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.073642 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.073694 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":83,"threads_started":1}
I20260928 12:29:55.073945 27035 tablet_service.cc:1467] Processing CreateTablet for tablet d52dd1ba61c942a9a7879398bf61ff48 (DEFAULT_TABLE table=test-table [id=b7a036b5930e45c59b2609b4759a558a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:55.074342 27035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52dd1ba61c942a9a7879398bf61ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:55.074409 27297 tablet_service.cc:1467] Processing CreateTablet for tablet d52dd1ba61c942a9a7879398bf61ff48 (DEFAULT_TABLE table=test-table [id=b7a036b5930e45c59b2609b4759a558a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:55.074544 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.068992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37169 (local address 127.0.0.1:58350)
0928 12:29:55.069389 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.069397 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:55.069436 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:55.069735 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:55.069738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:55.069741 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:55.069815 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.069817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.071799 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.071801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.072736 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.072739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.074289 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.074290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.074347 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.074350 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:55.074432 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 12:29:55.074474 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":6,"thread_start_us":130,"threads_started":1}
I20260928 12:29:55.074605 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.069240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.7.19.66:37169)
0928 12:29:55.069502 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.069506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.069515 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:29:55.069555 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.069557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.069573 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.069651 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:55.070760 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.071693 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.072842 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.074192 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.074285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.074370 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.074449 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.074484 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":94,"threads_started":1}
I20260928 12:29:55.074728 27297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52dd1ba61c942a9a7879398bf61ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:55.077316 27166 tablet_service.cc:1467] Processing CreateTablet for tablet d52dd1ba61c942a9a7879398bf61ff48 (DEFAULT_TABLE table=test-table [id=b7a036b5930e45c59b2609b4759a558a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 12:29:55.077628 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d52dd1ba61c942a9a7879398bf61ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:55.078080 27377 tablet_bootstrap.cc:492] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Bootstrap starting.
I20260928 12:29:55.078737 27377 tablet_bootstrap.cc:654] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:55.079087 27377 log.cc:824] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:55.079439 27378 tablet_bootstrap.cc:492] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Bootstrap starting.
I20260928 12:29:55.079809 27377 tablet_bootstrap.cc:492] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36: No bootstrap required, opened a new log
I20260928 12:29:55.079893 27377 ts_tablet_manager.cc:1397] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:55.080060 27378 tablet_bootstrap.cc:654] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:55.080367 27378 log.cc:824] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:55.081171 27381 tablet_bootstrap.cc:492] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Bootstrap starting.
I20260928 12:29:55.081314 27378 tablet_bootstrap.cc:492] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: No bootstrap required, opened a new log
I20260928 12:29:55.081408 27378 ts_tablet_manager.cc:1397] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 12:29:55.081897 27381 tablet_bootstrap.cc:654] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:55.082193 27381 log.cc:824] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:55.082521 27377 raft_consensus.cc:348] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.082670 27377 raft_consensus.cc:374] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:55.082716 27377 raft_consensus.cc:729] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fefb1fcdb95b49c7bfa8ac3f8a2a5f36, State: Initialized, Role: FOLLOWER
I20260928 12:29:55.082828 27377 consensus_queue.cc:260] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.082963 27381 tablet_bootstrap.cc:492] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: No bootstrap required, opened a new log
I20260928 12:29:55.083027 27381 ts_tablet_manager.cc:1397] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 12:29:55.083096 27377 ts_tablet_manager.cc:1428] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 12:29:55.083221 27101 heartbeater.cc:503] Master 127.7.19.126:41569 was elected leader, sending a full tablet report...
I20260928 12:29:55.083817 27378 raft_consensus.cc:348] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.083956 27378 raft_consensus.cc:374] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:55.083992 27378 raft_consensus.cc:729] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683336067a4b4558aba63e7578d22600, State: Initialized, Role: FOLLOWER
I20260928 12:29:55.084221 27378 consensus_queue.cc:260] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.084537 27378 ts_tablet_manager.cc:1428] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:29:55.084561 27363 heartbeater.cc:503] Master 127.7.19.126:41569 was elected leader, sending a full tablet report...
I20260928 12:29:55.085632 27381 raft_consensus.cc:348] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.085752 27381 raft_consensus.cc:374] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:55.085773 27381 raft_consensus.cc:729] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16a57ee5a174afdb241505b2410df43, State: Initialized, Role: FOLLOWER
I20260928 12:29:55.085870 27381 consensus_queue.cc:260] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.086078 27381 ts_tablet_manager.cc:1428] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 12:29:55.086187 27232 heartbeater.cc:503] Master 127.7.19.126:41569 was elected leader, sending a full tablet report...
I20260928 12:29:55.160252 27383 raft_consensus.cc:482] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 12:29:55.160372 27383 raft_consensus.cc:504] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.160761 27383 leader_election.cc:288] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 683336067a4b4558aba63e7578d22600 (127.7.19.67:35715), f16a57ee5a174afdb241505b2410df43 (127.7.19.66:37169)
I20260928 12:29:55.163790 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.161051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:39573 (local address 127.7.19.66:37169)
0928 12:29:55.161102 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.161104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.161116 (+ 12us) server_negotiation.cc:407] Connection header received
0928 12:29:55.161161 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.161162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.161172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.161239 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:55.161445 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.162070 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.163082 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.163450 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.163505 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.163514 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.163669 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.163718 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 12:29:55.163790 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.160948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.66:37169 (local address 127.7.19.65:39573)
0928 12:29:55.161008 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.161021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:55.161089 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:55.161253 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:55.161254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:55.161255 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:55.161350 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.161352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.162165 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.162166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.162991 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.162993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.163527 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.163528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.163596 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.163600 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 12:29:55.163661 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 12:29:55.163693 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":24}
I20260928 12:29:55.163864 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.161121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.19.65:41509 (local address 127.7.19.67:35715)
0928 12:29:55.161194 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.161200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.161309 (+ 109us) server_negotiation.cc:407] Connection header received
0928 12:29:55.161365 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.161366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.161376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.161431 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 12:29:55.161597 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.162100 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.163119 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.163466 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.163552 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.163560 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.163753 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.163792 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 12:29:55.163918 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.160948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.67:35715 (local address 127.7.19.65:41509)
0928 12:29:55.161243 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.161254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:29:55.161309 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:29:55.161443 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:29:55.161444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:29:55.161446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 12:29:55.161514 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.161516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.162240 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.162241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.163000 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:29:55.163002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.163572 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.163573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:29:55.163677 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.163680 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 12:29:55.163736 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 12:29:55.163771 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":107,"thread_start_us":109,"threads_started":1}
I20260928 12:29:55.164234 27317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52dd1ba61c942a9a7879398bf61ff48" candidate_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683336067a4b4558aba63e7578d22600" is_pre_election: true
I20260928 12:29:55.164215 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52dd1ba61c942a9a7879398bf61ff48" candidate_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f16a57ee5a174afdb241505b2410df43" is_pre_election: true
I20260928 12:29:55.164395 27186 raft_consensus.cc:2454] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fefb1fcdb95b49c7bfa8ac3f8a2a5f36 in term 0.
I20260928 12:29:55.164402 27317 raft_consensus.cc:2454] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fefb1fcdb95b49c7bfa8ac3f8a2a5f36 in term 0.
I20260928 12:29:55.164645 26989 leader_election.cc:302] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 683336067a4b4558aba63e7578d22600, fefb1fcdb95b49c7bfa8ac3f8a2a5f36; no voters:
I20260928 12:29:55.164901 27383 raft_consensus.cc:2781] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 12:29:55.165050 27383 raft_consensus.cc:482] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 12:29:55.165096 27383 raft_consensus.cc:3037] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:55.165824 27383 raft_consensus.cc:504] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.165973 27383 leader_election.cc:288] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [CANDIDATE]: Term 1 election: Requested vote from peers 683336067a4b4558aba63e7578d22600 (127.7.19.67:35715), f16a57ee5a174afdb241505b2410df43 (127.7.19.66:37169)
I20260928 12:29:55.166132 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52dd1ba61c942a9a7879398bf61ff48" candidate_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f16a57ee5a174afdb241505b2410df43"
I20260928 12:29:55.166126 27317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52dd1ba61c942a9a7879398bf61ff48" candidate_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683336067a4b4558aba63e7578d22600"
I20260928 12:29:55.166206 27317 raft_consensus.cc:3037] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:55.166210 27186 raft_consensus.cc:3037] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:55.166894 27186 raft_consensus.cc:2454] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fefb1fcdb95b49c7bfa8ac3f8a2a5f36 in term 1.
I20260928 12:29:55.166958 27317 raft_consensus.cc:2454] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fefb1fcdb95b49c7bfa8ac3f8a2a5f36 in term 1.
I20260928 12:29:55.167094 26990 leader_election.cc:302] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f16a57ee5a174afdb241505b2410df43, fefb1fcdb95b49c7bfa8ac3f8a2a5f36; no voters:
I20260928 12:29:55.167188 27383 raft_consensus.cc:2781] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:55.167338 27383 raft_consensus.cc:686] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [term 1 LEADER]: Becoming Leader. State: Replica: fefb1fcdb95b49c7bfa8ac3f8a2a5f36, State: Running, Role: LEADER
I20260928 12:29:55.167438 27383 consensus_queue.cc:237] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } }
I20260928 12:29:55.168219 26912 catalog_manager.cc:5495] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 reported cstate change: term changed from 0 to 1, leader changed from <none> to fefb1fcdb95b49c7bfa8ac3f8a2a5f36 (127.7.19.65). New cstate: current_term: 1 leader_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fefb1fcdb95b49c7bfa8ac3f8a2a5f36" member_type: VOTER last_known_addr { host: "127.7.19.65" port: 36013 } health_report { overall_health: HEALTHY } } }
W20260928 12:29:55.173931 27233 tablet.cc:2369] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 12:29:55.192564 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.190245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.7.19.65:36013)
0928 12:29:55.190312 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:55.190314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:55.190327 (+ 13us) server_negotiation.cc:407] Connection header received
0928 12:29:55.190364 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:55.190365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:55.190387 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:55.190462 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:29:55.190581 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.191056 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.191807 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:55.192017 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:55.192078 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:55.192247 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:29:55.192347 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:29:55.192424 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:55.192459 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 12:29:55.192477 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:29:55.194509 27186 raft_consensus.cc:1261] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 1 FOLLOWER]: Refusing update from remote peer fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:55.194566 27317 raft_consensus.cc:1261] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 1 FOLLOWER]: Refusing update from remote peer fefb1fcdb95b49c7bfa8ac3f8a2a5f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:29:55.194906 27383 consensus_queue.cc:1035] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f16a57ee5a174afdb241505b2410df43" member_type: VOTER last_known_addr { host: "127.7.19.66" port: 37169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:55.195590 27383 consensus_queue.cc:1035] T d52dd1ba61c942a9a7879398bf61ff48 P fefb1fcdb95b49c7bfa8ac3f8a2a5f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683336067a4b4558aba63e7578d22600" member_type: VOTER last_known_addr { host: "127.7.19.67" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:29:55.197451 27390 mvcc.cc:205] Tried to move back new op lower bound from 7334291845913772032 to 7334291845806325760. Current Snapshot: MvccSnapshot[applied={T|T < 7334291845913772032}]
I20260928 12:29:55.197844 27391 mvcc.cc:205] Tried to move back new op lower bound from 7334291845913772032 to 7334291845806325760. Current Snapshot: MvccSnapshot[applied={T|T < 7334291845913772032}]
I20260928 12:29:55.198153 27393 mvcc.cc:205] Tried to move back new op lower bound from 7334291845913772032 to 7334291845806325760. Current Snapshot: MvccSnapshot[applied={T|T < 7334291845913772032}]
I20260928 12:29:55.199769 26913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:55.199859 26913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 12:29:55.200708 26913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 280baeabda4f4cbf82ca05deb8a3307c: Sending DeleteTablet for 3 replicas of tablet d52dd1ba61c942a9a7879398bf61ff48
I20260928 12:29:55.201078 27166 tablet_service.cc:1514] Processing DeleteTablet for tablet d52dd1ba61c942a9a7879398bf61ff48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:55 UTC) from {username='slave'} at 127.0.0.1:58350
I20260928 12:29:55.201103 27297 tablet_service.cc:1514] Processing DeleteTablet for tablet d52dd1ba61c942a9a7879398bf61ff48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:55 UTC) from {username='slave'} at 127.0.0.1:43420
I20260928 12:29:55.201321 27399 tablet_replica.cc:330] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: stopping tablet replica
I20260928 12:29:55.201354 27035 tablet_service.cc:1514] Processing DeleteTablet for tablet d52dd1ba61c942a9a7879398bf61ff48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 12:29:55 UTC) from {username='slave'} at 127.0.0.1:37218
I20260928 12:29:55.201422 27399 raft_consensus.cc:2229] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:55.201520 27399 raft_consensus.cc:2258] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:55.201771 27400 tablet_replica.cc:330] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: stopping tablet replica
I20260928 12:29:55.201792 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26973
I20260928 12:29:55.201870 27400 raft_consensus.cc:2229] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 12:29:55.201963 27400 raft_consensus.cc:2258] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 12:29:55.203435 27399 ts_tablet_manager.cc:1905] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:55.204962 27400 ts_tablet_manager.cc:1905] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 12:29:55.205013 27399 ts_tablet_manager.cc:1918] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:55.205108 27399 log.cc:1196] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-1/wal/wals/d52dd1ba61c942a9a7879398bf61ff48
I20260928 12:29:55.205348 27399 ts_tablet_manager.cc:1939] T d52dd1ba61c942a9a7879398bf61ff48 P f16a57ee5a174afdb241505b2410df43: Deleting consensus metadata
I20260928 12:29:55.205744 26899 catalog_manager.cc:4841] TS f16a57ee5a174afdb241505b2410df43 (127.7.19.66:37169): tablet d52dd1ba61c942a9a7879398bf61ff48 (table test-table [id=b7a036b5930e45c59b2609b4759a558a]) successfully deleted
I20260928 12:29:55.206934 27400 ts_tablet_manager.cc:1918] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 12:29:55.207008 27400 log.cc:1196] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Deleting WAL directory at /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790598357699237-7245-0/minicluster-data/ts-2/wal/wals/d52dd1ba61c942a9a7879398bf61ff48
I20260928 12:29:55.207254 27400 ts_tablet_manager.cc:1939] T d52dd1ba61c942a9a7879398bf61ff48 P 683336067a4b4558aba63e7578d22600: Deleting consensus metadata
W20260928 12:29:55.207419 26897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.19.65:36013 (error 108)
I20260928 12:29:55.207708 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27104
I20260928 12:29:55.207760 26898 catalog_manager.cc:4841] TS 683336067a4b4558aba63e7578d22600 (127.7.19.67:35715): tablet d52dd1ba61c942a9a7879398bf61ff48 (table test-table [id=b7a036b5930e45c59b2609b4759a558a]) successfully deleted
I20260928 12:29:55.208279 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:55.207752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.65:36013 (local address 127.0.0.1:37228)
0928 12:29:55.208196 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 12:29:55.208242 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.65:36013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
W20260928 12:29:55.208823 26897 catalog_manager.cc:4584] TS fefb1fcdb95b49c7bfa8ac3f8a2a5f36 (127.7.19.65:36013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d52dd1ba61c942a9a7879398bf61ff48: Network error: Client connection negotiation failed: client connection to 127.7.19.65:36013: connect: Connection refused (error 111)
I20260928 12:29:55.212541 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27235
I20260928 12:29:55.217085 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 26882
2026-09-28T12:29:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (712 ms)
[----------] 6 tests from EncryptionPolicyTest (8337 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:55 dist-test-slave-cb7x krb5kdc[27405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:55 dist-test-slave-cb7x krb5kdc[27405](info): set up 2 sockets
Sep 28 12:29:55 dist-test-slave-cb7x krb5kdc[27405](info): commencing operation
krb5kdc: starting...
W20260928 12:29:56.248646 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:29:56 dist-test-slave-cb7x krb5kdc[27405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:29:56.365964 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:56.445550 27418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:56.445808 27418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:56.445847 27418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:56.447037 27418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:56.447111 27418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:56.447134 27418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:56.447154 27418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:56.447207 27418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:56.448693 27418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:56.448976 27418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:56.449208 27418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:56.454216 27423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:56.454216 27426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:56.454233 27418 server_base.cc:1034] running on GCE node
W20260928 12:29:56.454241 27424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:56.454710 27418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:56.455004 27418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:56.456182 27418 hybrid_clock.cc:648] HybridClock initialized: now 1790598596456177 us; error 34 us; skew 500 ppm
Sep 28 12:29:56 dist-test-slave-cb7x krb5kdc[27405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:56.458675 27418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:29:56.459117 27418 webserver.cc:460] Webserver started at http://127.0.0.1:34633/ using document root <none> and password file <none>
I20260928 12:29:56.459306 27418 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:56.459344 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:56.459429 27418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:56.460425 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "e919c103835a48c28ac6594c85e1448d"
format_stamp: "Formatted at 2026-09-28 12:29:56 on dist-test-slave-cb7x"
server_key: "db487a1d07fd4971d27a3d5f91bfb20f"
server_key_iv: "82f354a2a9b1fa1377e32208584b47fd"
server_key_version: "encryptionkey@0"
I20260928 12:29:56.460798 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "e919c103835a48c28ac6594c85e1448d"
format_stamp: "Formatted at 2026-09-28 12:29:56 on dist-test-slave-cb7x"
server_key: "db487a1d07fd4971d27a3d5f91bfb20f"
server_key_iv: "82f354a2a9b1fa1377e32208584b47fd"
server_key_version: "encryptionkey@0"
I20260928 12:29:56.462024 27418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:56.462733 27433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:56.462911 27418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:56.462996 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "e919c103835a48c28ac6594c85e1448d"
format_stamp: "Formatted at 2026-09-28 12:29:56 on dist-test-slave-cb7x"
server_key: "db487a1d07fd4971d27a3d5f91bfb20f"
server_key_iv: "82f354a2a9b1fa1377e32208584b47fd"
server_key_version: "encryptionkey@0"
I20260928 12:29:56.463052 27418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:56.473884 27418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:56.476145 27418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:56.476310 27418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:56.480149 27418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33057
I20260928 12:29:56.480225 27485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33057 every 8 connection(s)
I20260928 12:29:56.480592 27418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:56.481627 27486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:56.483846 27486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d: Bootstrap starting.
I20260928 12:29:56.484465 27486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:56.484822 27486 log.cc:824] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:56.485548 27486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d: No bootstrap required, opened a new log
I20260928 12:29:56.487089 27486 raft_consensus.cc:348] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e919c103835a48c28ac6594c85e1448d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260928 12:29:56.487246 27486 raft_consensus.cc:374] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:56.487278 27486 raft_consensus.cc:729] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e919c103835a48c28ac6594c85e1448d, State: Initialized, Role: FOLLOWER
I20260928 12:29:56.487445 27486 consensus_queue.cc:260] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e919c103835a48c28ac6594c85e1448d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260928 12:29:56.487511 27486 raft_consensus.cc:388] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:56.487532 27486 raft_consensus.cc:482] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:56.487579 27486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:56.488112 27486 raft_consensus.cc:504] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e919c103835a48c28ac6594c85e1448d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260928 12:29:56.488196 27486 leader_election.cc:302] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e919c103835a48c28ac6594c85e1448d; no voters:
I20260928 12:29:56.488387 27486 leader_election.cc:288] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:56.488478 27489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:56.488634 27486 sys_catalog.cc:564] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:56.488703 27489 raft_consensus.cc:686] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [term 1 LEADER]: Becoming Leader. State: Replica: e919c103835a48c28ac6594c85e1448d, State: Running, Role: LEADER
I20260928 12:29:56.488796 27489 consensus_queue.cc:237] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e919c103835a48c28ac6594c85e1448d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260928 12:29:56.489178 27490 sys_catalog.cc:455] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e919c103835a48c28ac6594c85e1448d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e919c103835a48c28ac6594c85e1448d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260928 12:29:56.489270 27490 sys_catalog.cc:458] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:56.489313 27491 sys_catalog.cc:455] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e919c103835a48c28ac6594c85e1448d. Latest consensus state: current_term: 1 leader_uuid: "e919c103835a48c28ac6594c85e1448d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e919c103835a48c28ac6594c85e1448d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260928 12:29:56.489388 27491 sys_catalog.cc:458] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:56.489454 27497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:56.489961 27497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:56.490456 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27418
I20260928 12:29:56.490576 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:56.490746 7245 external_mini_cluster.cc:1428] Setting key f16250372dd7635bf8501775bb959825
I20260928 12:29:56.492268 27497 catalog_manager.cc:1347] Generated new cluster ID: caab1f0a23314724a59400da70f8277a
I20260928 12:29:56.492321 27497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:29:56 dist-test-slave-cb7x krb5kdc[27405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:29:56.500502 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:56.491271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55707 (local address 127.0.0.1:33057)
0928 12:29:56.491488 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:56.491496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:56.491529 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:29:56.491639 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:56.491646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:56.491709 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:56.491852 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:56.492606 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:56.493592 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:56.494358 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:56.494795 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:56.494909 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:56.497285 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:56.497289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:56.497289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:56.497301 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:56.498951 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:56.499353 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:56.499358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:56.499358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:56.499460 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:56.499695 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:56.499696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:56.499696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:56.499930 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:56.500113 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:56.500196 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 12:29:56.500301 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260928 12:29:56.501452 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:29:56.503736 27497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:56.504415 27497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:56.509439 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:56.502777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53767 (local address 127.0.0.1:33057)
0928 12:29:56.502885 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:56.502888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:56.503050 (+ 162us) server_negotiation.cc:407] Connection header received
0928 12:29:56.503112 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:56.503113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:56.503124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:56.503167 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:56.504003 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:56.504533 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:56.505310 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:56.505524 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:56.505598 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:56.506740 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:56.506741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:56.506741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:29:56.506748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:56.508174 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:56.508542 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:56.508543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:56.508543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:56.508600 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:29:56.508889 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:29:56.508890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:29:56.508890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:29:56.508998 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:56.509104 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:56.509238 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 12:29:56.509329 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 12:29:56.510661 27497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e919c103835a48c28ac6594c85e1448d: Generated new TSK 0
I20260928 12:29:56.510841 27497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:29:56.514230 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27418
W20260928 12:29:56.518428 27403 connection.cc:535] client connection to 127.0.0.1:33057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33057): Connection reset by peer (error 104)
2026-09-28T12:29:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:56 dist-test-slave-cb7x krb5kdc[27517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:56 dist-test-slave-cb7x krb5kdc[27517](info): set up 2 sockets
Sep 28 12:29:56 dist-test-slave-cb7x krb5kdc[27517](info): commencing operation
krb5kdc: starting...
W20260928 12:29:57.556950 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:29:57 dist-test-slave-cb7x krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:29:57.676724 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:57.760150 27530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:57.760393 27530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:57.760432 27530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:57.761639 27530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:57.761708 27530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:57.761731 27530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:57.761786 27530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:57.761817 27530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:57.763235 27530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:57.763505 27530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:57.763723 27530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:57.768550 27535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:57.768558 27536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:57.768697 27530 server_base.cc:1034] running on GCE node
W20260928 12:29:57.768833 27538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:57.769100 27530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:57.769400 27530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:57.770578 27530 hybrid_clock.cc:648] HybridClock initialized: now 1790598597770555 us; error 42 us; skew 500 ppm
Sep 28 12:29:57 dist-test-slave-cb7x krb5kdc[27517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:57.782446 27530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:29:57.782948 27530 webserver.cc:460] Webserver started at http://127.0.0.1:36199/ using document root <none> and password file <none>
I20260928 12:29:57.783226 27530 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:57.783303 27530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:57.783463 27530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:57.784638 27530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "9d393551cbf24653bc1fa5694e0dd057"
format_stamp: "Formatted at 2026-09-28 12:29:57 on dist-test-slave-cb7x"
server_key: "12628695637a9cd388a31b911d753f4f"
server_key_iv: "a399be8c02477774b63e32e8587dd0fb"
server_key_version: "encryptionkey@0"
I20260928 12:29:57.785104 27530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "9d393551cbf24653bc1fa5694e0dd057"
format_stamp: "Formatted at 2026-09-28 12:29:57 on dist-test-slave-cb7x"
server_key: "12628695637a9cd388a31b911d753f4f"
server_key_iv: "a399be8c02477774b63e32e8587dd0fb"
server_key_version: "encryptionkey@0"
I20260928 12:29:57.786552 27530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:57.787336 27545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:57.787560 27530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:57.787675 27530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "9d393551cbf24653bc1fa5694e0dd057"
format_stamp: "Formatted at 2026-09-28 12:29:57 on dist-test-slave-cb7x"
server_key: "12628695637a9cd388a31b911d753f4f"
server_key_iv: "a399be8c02477774b63e32e8587dd0fb"
server_key_version: "encryptionkey@0"
I20260928 12:29:57.787750 27530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:57.826830 27530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:57.829213 27530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:57.829385 27530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:57.833109 27530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34897
I20260928 12:29:57.833158 27597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34897 every 8 connection(s)
I20260928 12:29:57.833528 27530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:57.834491 27598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:57.836804 27598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057: Bootstrap starting.
I20260928 12:29:57.837399 27598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:57.837698 27598 log.cc:824] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:57.838371 27598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057: No bootstrap required, opened a new log
I20260928 12:29:57.839934 27598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d393551cbf24653bc1fa5694e0dd057" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260928 12:29:57.840073 27598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:57.840095 27598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d393551cbf24653bc1fa5694e0dd057, State: Initialized, Role: FOLLOWER
I20260928 12:29:57.840291 27598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d393551cbf24653bc1fa5694e0dd057" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260928 12:29:57.840364 27598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:57.840387 27598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:57.840451 27598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:57.841004 27598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d393551cbf24653bc1fa5694e0dd057" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260928 12:29:57.841111 27598 leader_election.cc:302] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d393551cbf24653bc1fa5694e0dd057; no voters:
I20260928 12:29:57.841280 27598 leader_election.cc:288] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:57.841313 27601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:57.841475 27601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [term 1 LEADER]: Becoming Leader. State: Replica: 9d393551cbf24653bc1fa5694e0dd057, State: Running, Role: LEADER
I20260928 12:29:57.841547 27598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:57.841620 27601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d393551cbf24653bc1fa5694e0dd057" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260928 12:29:57.841760 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27530
I20260928 12:29:57.841921 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:57.841995 27602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d393551cbf24653bc1fa5694e0dd057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d393551cbf24653bc1fa5694e0dd057" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } } }
I20260928 12:29:57.842093 27602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:57.842144 7245 external_mini_cluster.cc:1428] Setting key 3848acbf4950b6f9a28931bb375f1565
I20260928 12:29:57.842332 27605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:57.842422 27603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d393551cbf24653bc1fa5694e0dd057. Latest consensus state: current_term: 1 leader_uuid: "9d393551cbf24653bc1fa5694e0dd057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d393551cbf24653bc1fa5694e0dd057" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } } }
I20260928 12:29:57.842502 27603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057 [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:57.842890 27605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:57.844455 27605 catalog_manager.cc:1347] Generated new cluster ID: 70d68190aaf94d5baa01650e24efd383
I20260928 12:29:57.844507 27605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:57.849090 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:57.842602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33413 (local address 127.0.0.1:34897)
0928 12:29:57.843576 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:57.843584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:57.844185 (+ 601us) server_negotiation.cc:407] Connection header received
0928 12:29:57.844401 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:57.844409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:57.844466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:57.844615 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:57.844978 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:57.846300 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:57.847057 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:57.847648 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:57.847744 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:57.848403 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:57.848406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:57.848407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:57.848601 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:57.848640 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:57.848653 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:57.848762 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:57.848869 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 12:29:57.848894 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":76,"threads_started":1}
I20260928 12:29:57.850088 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:29:57.852427 27605 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:57.853138 27605 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:57.855693 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:57.852141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33603 (local address 127.0.0.1:34897)
0928 12:29:57.852176 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:57.852177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:57.852347 (+ 170us) server_negotiation.cc:407] Connection header received
0928 12:29:57.852441 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:57.852442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:57.852453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:57.852484 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:57.852692 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:57.853371 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:57.854364 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:29:57.854694 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:29:57.854730 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:29:57.855454 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:57.855455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:57.855455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:57.855459 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:57.855471 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:57.855475 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:57.855503 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:57.855613 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 12:29:57.855625 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:29:57.861577 27605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d393551cbf24653bc1fa5694e0dd057: Generated new TSK 0
I20260928 12:29:57.861749 27605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:29:57.867775 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27530
W20260928 12:29:57.871450 27515 connection.cc:535] client connection to 127.0.0.1:34897 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34897): Connection reset by peer (error 104)
2026-09-28T12:29:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:57 dist-test-slave-cb7x krb5kdc[27629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:57 dist-test-slave-cb7x krb5kdc[27629](info): set up 2 sockets
Sep 28 12:29:57 dist-test-slave-cb7x krb5kdc[27629](info): commencing operation
krb5kdc: starting...
W20260928 12:29:59.000739 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:29:59 dist-test-slave-cb7x krb5kdc[27629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:29:59.123842 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:29:59.203297 27642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:29:59.203517 27642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:29:59.203555 27642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:29:59.204748 27642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:29:59.204804 27642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:29:59.204828 27642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:29:59.204854 27642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:29:59.204879 27642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:29:59.206296 27642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:29:59.206559 27642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:29:59.206779 27642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:29:59.211525 27648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:59.211534 27647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:29:59.211649 27650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:29:59.211720 27642 server_base.cc:1034] running on GCE node
I20260928 12:29:59.211961 27642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:29:59.212203 27642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:29:59.213359 27642 hybrid_clock.cc:648] HybridClock initialized: now 1790598599213337 us; error 37 us; skew 500 ppm
Sep 28 12:29:59 dist-test-slave-cb7x krb5kdc[27629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:29:59.215828 27642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:29:59.216254 27642 webserver.cc:460] Webserver started at http://127.0.0.1:36783/ using document root <none> and password file <none>
I20260928 12:29:59.216428 27642 fs_manager.cc:359] Metadata directory not provided
I20260928 12:29:59.216467 27642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:29:59.216550 27642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:29:59.217587 27642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "0afbd17a1ba54460a3aa46e5395d459c"
format_stamp: "Formatted at 2026-09-28 12:29:59 on dist-test-slave-cb7x"
server_key: "cfe5b2f21b0d145c085729bced282e46"
server_key_iv: "e1308fbecf8ed207d29031b149b1f09b"
server_key_version: "encryptionkey@0"
I20260928 12:29:59.217873 27642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "0afbd17a1ba54460a3aa46e5395d459c"
format_stamp: "Formatted at 2026-09-28 12:29:59 on dist-test-slave-cb7x"
server_key: "cfe5b2f21b0d145c085729bced282e46"
server_key_iv: "e1308fbecf8ed207d29031b149b1f09b"
server_key_version: "encryptionkey@0"
I20260928 12:29:59.219074 27642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:29:59.219759 27657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:29:59.220113 27642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:29:59.220211 27642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "0afbd17a1ba54460a3aa46e5395d459c"
format_stamp: "Formatted at 2026-09-28 12:29:59 on dist-test-slave-cb7x"
server_key: "cfe5b2f21b0d145c085729bced282e46"
server_key_iv: "e1308fbecf8ed207d29031b149b1f09b"
server_key_version: "encryptionkey@0"
I20260928 12:29:59.220284 27642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:29:59.223551 27642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:29:59.225446 27642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:29:59.225589 27642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:29:59.229663 27642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41901
I20260928 12:29:59.229692 27709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41901 every 8 connection(s)
I20260928 12:29:59.230021 27642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:29:59.230955 27710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:29:59.233232 27710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c: Bootstrap starting.
I20260928 12:29:59.233752 27710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c: Neither blocks nor log segments found. Creating new log.
I20260928 12:29:59.233991 27710 log.cc:824] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c: Log is configured to *not* fsync() on all Append() calls
I20260928 12:29:59.234654 27710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c: No bootstrap required, opened a new log
I20260928 12:29:59.236157 27710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbd17a1ba54460a3aa46e5395d459c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260928 12:29:59.236292 27710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:29:59.236315 27710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afbd17a1ba54460a3aa46e5395d459c, State: Initialized, Role: FOLLOWER
I20260928 12:29:59.236450 27710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbd17a1ba54460a3aa46e5395d459c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260928 12:29:59.236527 27710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:29:59.236550 27710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:29:59.236577 27710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:29:59.237102 27710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbd17a1ba54460a3aa46e5395d459c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260928 12:29:59.237188 27710 leader_election.cc:302] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afbd17a1ba54460a3aa46e5395d459c; no voters:
I20260928 12:29:59.237305 27710 leader_election.cc:288] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:29:59.237377 27713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:29:59.237574 27713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [term 1 LEADER]: Becoming Leader. State: Replica: 0afbd17a1ba54460a3aa46e5395d459c, State: Running, Role: LEADER
I20260928 12:29:59.237602 27710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:29:59.237723 27713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbd17a1ba54460a3aa46e5395d459c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } }
I20260928 12:29:59.238157 27715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afbd17a1ba54460a3aa46e5395d459c. Latest consensus state: current_term: 1 leader_uuid: "0afbd17a1ba54460a3aa46e5395d459c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbd17a1ba54460a3aa46e5395d459c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } } }
I20260928 12:29:59.238292 27715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:59.238440 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27642
I20260928 12:29:59.238561 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:29:59.238723 27722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:29:59.238770 7245 external_mini_cluster.cc:1428] Setting key e5cf98d831273e76227d0396c702046c
I20260928 12:29:59.239056 27714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0afbd17a1ba54460a3aa46e5395d459c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbd17a1ba54460a3aa46e5395d459c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41901 } } }
I20260928 12:29:59.239200 27714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c [sys.catalog]: This master's current role is: LEADER
I20260928 12:29:59.239240 27722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:29:59.241137 27722 catalog_manager.cc:1347] Generated new cluster ID: 698bb615bed5459d85259b32eb171a3a
I20260928 12:29:59.241179 27722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:29:59.245101 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:59.239450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57727 (local address 127.0.0.1:41901)
0928 12:29:59.239625 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:59.239634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:59.239857 (+ 223us) server_negotiation.cc:407] Connection header received
0928 12:29:59.239995 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:59.240003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:59.240040 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:59.240184 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:59.240923 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:59.240926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:59.240927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:59.240931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:59.240998 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:59.241015 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:59.244787 (+ 3772us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:59.244886 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:29:59.244929 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 12:29:59.245718 27722 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:29:59.245898 27722 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:29:59.246203 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:29:59.248730 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:29:59.247575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44163 (local address 127.0.0.1:41901)
0928 12:29:59.247610 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:29:59.247612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:29:59.247738 (+ 126us) server_negotiation.cc:407] Connection header received
0928 12:29:59.247799 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:29:59.247800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:29:59.247806 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:29:59.247860 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:29:59.248472 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:29:59.248474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:29:59.248474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:29:59.248479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:29:59.248494 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:29:59.248498 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:29:59.248555 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 12:29:59.248617 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 12:29:59.248630 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:29:59.250394 27722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0afbd17a1ba54460a3aa46e5395d459c: Generated new TSK 0
I20260928 12:29:59.250571 27722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:29:59.255920 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27642
2026-09-28T12:29:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:29:59 dist-test-slave-cb7x krb5kdc[27741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:29:59 dist-test-slave-cb7x krb5kdc[27741](info): set up 2 sockets
Sep 28 12:29:59 dist-test-slave-cb7x krb5kdc[27741](info): commencing operation
krb5kdc: starting...
W20260928 12:30:00.300649 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:00 dist-test-slave-cb7x krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:00.419720 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:00.502018 27754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:00.502256 27754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:00.502301 27754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:00.503456 27754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:00.503528 27754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:00.503552 27754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:00.503573 27754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:00.503604 27754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:00.505162 27754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:00.505434 27754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:00.505692 27754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:00.510433 27759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:00.510538 27754 server_base.cc:1034] running on GCE node
W20260928 12:30:00.510442 27760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:00.510452 27762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:00.510921 27754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:00.511257 27754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:00.512425 27754 hybrid_clock.cc:648] HybridClock initialized: now 1790598600512422 us; error 36 us; skew 500 ppm
Sep 28 12:30:00 dist-test-slave-cb7x krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:00.515075 27754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:00.515529 27754 webserver.cc:460] Webserver started at http://127.0.0.1:36539/ using document root <none> and password file <none>
I20260928 12:30:00.515760 27754 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:00.515829 27754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:00.515985 27754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:00.517117 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "0a7ca302ce5e43d79d9313e1c7f55176"
format_stamp: "Formatted at 2026-09-28 12:30:00 on dist-test-slave-cb7x"
server_key: "f5d290ea3abbd895b0808adba32bf93f"
server_key_iv: "eeb96f9591089f3d740562ca11893418"
server_key_version: "encryptionkey@0"
I20260928 12:30:00.517481 27754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "0a7ca302ce5e43d79d9313e1c7f55176"
format_stamp: "Formatted at 2026-09-28 12:30:00 on dist-test-slave-cb7x"
server_key: "f5d290ea3abbd895b0808adba32bf93f"
server_key_iv: "eeb96f9591089f3d740562ca11893418"
server_key_version: "encryptionkey@0"
I20260928 12:30:00.518865 27754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:00.519588 27769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:00.519757 27754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:00.519860 27754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "0a7ca302ce5e43d79d9313e1c7f55176"
format_stamp: "Formatted at 2026-09-28 12:30:00 on dist-test-slave-cb7x"
server_key: "f5d290ea3abbd895b0808adba32bf93f"
server_key_iv: "eeb96f9591089f3d740562ca11893418"
server_key_version: "encryptionkey@0"
I20260928 12:30:00.519944 27754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:00.531222 27754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:00.533533 27754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:00.533699 27754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:00.537595 27754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33249
I20260928 12:30:00.537645 27821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33249 every 8 connection(s)
I20260928 12:30:00.538000 27754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:00.538934 27822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:00.541252 27822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176: Bootstrap starting.
I20260928 12:30:00.541839 27822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:00.542124 27822 log.cc:824] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:00.542804 27822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176: No bootstrap required, opened a new log
I20260928 12:30:00.544425 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27754
I20260928 12:30:00.544382 27822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260928 12:30:00.544504 27822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:00.544574 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:00.544575 27822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7ca302ce5e43d79d9313e1c7f55176, State: Initialized, Role: FOLLOWER
I20260928 12:30:00.544749 27822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260928 12:30:00.544786 7245 external_mini_cluster.cc:1428] Setting key dff8bac01091f2bf9aaaa0f18901d315
I20260928 12:30:00.544818 27822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:00.544872 27822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:00.544924 27822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:00.546010 27822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260928 12:30:00.546121 27822 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7ca302ce5e43d79d9313e1c7f55176; no voters:
I20260928 12:30:00.546303 27822 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:00.546322 27827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:00.546437 27827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7ca302ce5e43d79d9313e1c7f55176, State: Running, Role: LEADER
I20260928 12:30:00.546553 27822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:00.546581 27827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260928 12:30:00.547016 27827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7ca302ce5e43d79d9313e1c7f55176. Latest consensus state: current_term: 1 leader_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } } }
I20260928 12:30:00.547276 27827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:00.547187 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7ca302ce5e43d79d9313e1c7f55176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } } }
I20260928 12:30:00.547689 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:00.547871 27836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:00.548588 27836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:00.550100 27836 catalog_manager.cc:1347] Generated new cluster ID: 2d1bc3babdf24ac8b5b821eb7dba8de0
I20260928 12:30:00.550155 27836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:00 dist-test-slave-cb7x krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:30:00.556348 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:00.545295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59207 (local address 127.0.0.1:33249)
0928 12:30:00.545530 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:00.545544 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:00.545584 (+ 40us) server_negotiation.cc:407] Connection header received
0928 12:30:00.545728 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:00.545735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:00.545801 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:00.545948 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:00.546728 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:00.548216 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:00.549232 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:00.549803 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:00.549936 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:00.552606 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:00.552610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:00.552610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:00.552622 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:00.554637 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:00.555022 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:00.555025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:00.555025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:00.555167 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:00.555396 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:00.555397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:00.555397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:00.555757 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:00.555923 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:00.555995 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:30:00.556132 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":118,"threads_started":1}
I20260928 12:30:00.557578 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:00.566229 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:00.559010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40095 (local address 127.0.0.1:33249)
0928 12:30:00.559044 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:00.559046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:00.559218 (+ 172us) server_negotiation.cc:407] Connection header received
0928 12:30:00.559311 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:00.559312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:00.559322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:00.559349 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:00.560038 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:00.560853 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:00.561832 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:00.562159 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:00.562239 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:00.563520 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:00.563522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:00.563522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:00.563529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:00.565108 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:00.565480 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:00.565481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:00.565481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:00.565538 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:00.565731 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:00.565731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:00.565732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:00.565855 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:00.565961 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:00.566060 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 12:30:00.566132 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:30:00.571198 27836 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:00.571695 27836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:00.574990 27836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a7ca302ce5e43d79d9313e1c7f55176: Generated new TSK 0
I20260928 12:30:00.575171 27836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:00.583936 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27754
2026-09-28T12:30:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:00 dist-test-slave-cb7x krb5kdc[27853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:00 dist-test-slave-cb7x krb5kdc[27853](info): set up 2 sockets
Sep 28 12:30:00 dist-test-slave-cb7x krb5kdc[27853](info): commencing operation
krb5kdc: starting...
W20260928 12:30:01.616561 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:01 dist-test-slave-cb7x krb5kdc[27853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:01.736649 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:01.815586 27866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:01.815816 27866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:01.815853 27866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:01.817047 27866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:01.817109 27866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:01.817133 27866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:01.817163 27866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:01.817195 27866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:01.818640 27866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:01.818899 27866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:01.819123 27866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:01.823889 27871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:01.823923 27872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:01.823942 27874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:01.823980 27866 server_base.cc:1034] running on GCE node
I20260928 12:30:01.824369 27866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:01.824702 27866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:01.825862 27866 hybrid_clock.cc:648] HybridClock initialized: now 1790598601825850 us; error 30 us; skew 500 ppm
Sep 28 12:30:01 dist-test-slave-cb7x krb5kdc[27853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:01.828178 27866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:01.828727 27866 webserver.cc:460] Webserver started at http://127.0.0.1:39499/ using document root <none> and password file <none>
I20260928 12:30:01.828963 27866 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:01.829028 27866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:01.829162 27866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:01.830186 27866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "8ce33291b95e4f8aa1713fcc9467826e"
format_stamp: "Formatted at 2026-09-28 12:30:01 on dist-test-slave-cb7x"
server_key: "afe1b011b5b7af9566b4eb308da01148"
server_key_iv: "df1ff4ff7c15aeb01a8590007305e02a"
server_key_version: "encryptionkey@0"
I20260928 12:30:01.830539 27866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "8ce33291b95e4f8aa1713fcc9467826e"
format_stamp: "Formatted at 2026-09-28 12:30:01 on dist-test-slave-cb7x"
server_key: "afe1b011b5b7af9566b4eb308da01148"
server_key_iv: "df1ff4ff7c15aeb01a8590007305e02a"
server_key_version: "encryptionkey@0"
I20260928 12:30:01.831866 27866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:01.832626 27881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:01.832849 27866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:01.832947 27866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "8ce33291b95e4f8aa1713fcc9467826e"
format_stamp: "Formatted at 2026-09-28 12:30:01 on dist-test-slave-cb7x"
server_key: "afe1b011b5b7af9566b4eb308da01148"
server_key_iv: "df1ff4ff7c15aeb01a8590007305e02a"
server_key_version: "encryptionkey@0"
I20260928 12:30:01.833032 27866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:01.855769 27866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:01.858064 27866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:01.858248 27866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:01.862408 27866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39005
I20260928 12:30:01.862444 27933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39005 every 8 connection(s)
I20260928 12:30:01.862815 27866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:01.863775 27934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:01.866153 27934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e: Bootstrap starting.
I20260928 12:30:01.866689 27934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:01.866916 27934 log.cc:824] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:01.867535 27934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e: No bootstrap required, opened a new log
I20260928 12:30:01.869066 27934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce33291b95e4f8aa1713fcc9467826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39005 } }
I20260928 12:30:01.869228 27934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:01.869253 27934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce33291b95e4f8aa1713fcc9467826e, State: Initialized, Role: FOLLOWER
I20260928 12:30:01.869387 27934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce33291b95e4f8aa1713fcc9467826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39005 } }
I20260928 12:30:01.869472 27934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:01.869493 27934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:01.869521 27934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:01.869982 27934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce33291b95e4f8aa1713fcc9467826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39005 } }
I20260928 12:30:01.870062 27934 leader_election.cc:302] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ce33291b95e4f8aa1713fcc9467826e; no voters:
I20260928 12:30:01.870187 27934 leader_election.cc:288] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:01.870263 27937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:01.870441 27937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [term 1 LEADER]: Becoming Leader. State: Replica: 8ce33291b95e4f8aa1713fcc9467826e, State: Running, Role: LEADER
I20260928 12:30:01.870456 27934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:01.870529 27937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce33291b95e4f8aa1713fcc9467826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39005 } }
I20260928 12:30:01.870926 27939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ce33291b95e4f8aa1713fcc9467826e. Latest consensus state: current_term: 1 leader_uuid: "8ce33291b95e4f8aa1713fcc9467826e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce33291b95e4f8aa1713fcc9467826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39005 } } }
I20260928 12:30:01.871069 27939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:01.871292 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27866
I20260928 12:30:01.871410 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:01.871485 27946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:01.871632 7245 external_mini_cluster.cc:1428] Setting key 85cb9a3b9f9d85bf4c9ec11aa78a3b62
I20260928 12:30:01.871055 27938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ce33291b95e4f8aa1713fcc9467826e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ce33291b95e4f8aa1713fcc9467826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39005 } } }
I20260928 12:30:01.871986 27938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:01.872008 27946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:01.873780 27946 catalog_manager.cc:1347] Generated new cluster ID: d1158d2ee09347d09b107b73776c7029
I20260928 12:30:01.873834 27946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:01.878170 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:01.872168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44279 (local address 127.0.0.1:39005)
0928 12:30:01.872448 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:01.872456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:01.872485 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:30:01.872796 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:01.872804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:01.872867 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:01.873347 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:01.873461 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:01.874831 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:01.875985 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:01.876585 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:01.876706 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:01.877409 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:01.877412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:01.877413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:01.877625 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:01.877660 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:01.877674 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:01.877734 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:01.877830 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 12:30:01.877975 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
I20260928 12:30:01.879324 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:01.887674 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:01.884337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.0.0.1:39005)
0928 12:30:01.884371 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:01.884373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:01.884434 (+ 61us) server_negotiation.cc:407] Connection header received
0928 12:30:01.884489 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:01.884490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:01.884512 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:01.884580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:01.884687 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:01.885432 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:01.886382 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:01.886695 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:01.886764 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:01.887427 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:01.887428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:01.887428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:01.887432 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:01.887447 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:01.887451 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:01.887509 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:01.887568 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:30:01.887595 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:30:01.897671 27946 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:01.898177 27946 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:01.911149 27946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ce33291b95e4f8aa1713fcc9467826e: Generated new TSK 0
I20260928 12:30:01.911365 27946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:01.918773 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27866
2026-09-28T12:30:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:01 dist-test-slave-cb7x krb5kdc[27965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:01 dist-test-slave-cb7x krb5kdc[27965](info): set up 2 sockets
Sep 28 12:30:01 dist-test-slave-cb7x krb5kdc[27965](info): commencing operation
krb5kdc: starting...
W20260928 12:30:02.944783 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:03 dist-test-slave-cb7x krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:03.061769 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:03.140362 27978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:03.140561 27978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:03.140583 27978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:03.141728 27978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:03.141773 27978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:03.141788 27978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:03.141800 27978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:03.141840 27978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:03.143769 27978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:03.144006 27978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:03.144222 27978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:30:03.149279 27978 server_base.cc:1034] running on GCE node
W20260928 12:30:03.149299 27983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:03.149358 27984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:03.149298 27986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:03.149660 27978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:03.149940 27978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:03.151115 27978 hybrid_clock.cc:648] HybridClock initialized: now 1790598603151107 us; error 30 us; skew 500 ppm
Sep 28 12:30:03 dist-test-slave-cb7x krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:03.162104 27978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:03.162626 27978 webserver.cc:460] Webserver started at http://127.0.0.1:40157/ using document root <none> and password file <none>
I20260928 12:30:03.162864 27978 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:03.162933 27978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:03.163089 27978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:03.164165 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "1c6bf06d793348dcbc915dbded9ec5af"
format_stamp: "Formatted at 2026-09-28 12:30:03 on dist-test-slave-cb7x"
server_key: "03c458bb248c2bb06d1d890291138f0a"
server_key_iv: "03a6aecf1649c2593c94d569e6eeb81e"
server_key_version: "encryptionkey@0"
I20260928 12:30:03.164541 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "1c6bf06d793348dcbc915dbded9ec5af"
format_stamp: "Formatted at 2026-09-28 12:30:03 on dist-test-slave-cb7x"
server_key: "03c458bb248c2bb06d1d890291138f0a"
server_key_iv: "03a6aecf1649c2593c94d569e6eeb81e"
server_key_version: "encryptionkey@0"
I20260928 12:30:03.165941 27978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:03.166707 27993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:03.166908 27978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:03.166994 27978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "1c6bf06d793348dcbc915dbded9ec5af"
format_stamp: "Formatted at 2026-09-28 12:30:03 on dist-test-slave-cb7x"
server_key: "03c458bb248c2bb06d1d890291138f0a"
server_key_iv: "03a6aecf1649c2593c94d569e6eeb81e"
server_key_version: "encryptionkey@0"
I20260928 12:30:03.167047 27978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:03.170390 27978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:03.172573 27978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:03.172741 27978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:03.176512 27978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36839
I20260928 12:30:03.176556 28045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36839 every 8 connection(s)
I20260928 12:30:03.176900 27978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:03.177845 28046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:03.179957 28046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af: Bootstrap starting.
I20260928 12:30:03.180537 28046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:03.180877 28046 log.cc:824] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:03.181536 28046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af: No bootstrap required, opened a new log
I20260928 12:30:03.183060 28046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6bf06d793348dcbc915dbded9ec5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260928 12:30:03.183214 28046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:03.183290 28046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6bf06d793348dcbc915dbded9ec5af, State: Initialized, Role: FOLLOWER
I20260928 12:30:03.183454 28046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6bf06d793348dcbc915dbded9ec5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260928 12:30:03.183537 28046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:03.183578 28046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:03.183626 28046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:03.184118 28046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6bf06d793348dcbc915dbded9ec5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260928 12:30:03.184230 28046 leader_election.cc:302] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c6bf06d793348dcbc915dbded9ec5af; no voters:
I20260928 12:30:03.184409 28046 leader_election.cc:288] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:03.184442 28049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:03.184603 28049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [term 1 LEADER]: Becoming Leader. State: Replica: 1c6bf06d793348dcbc915dbded9ec5af, State: Running, Role: LEADER
I20260928 12:30:03.184717 28046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:03.184759 28049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6bf06d793348dcbc915dbded9ec5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } }
I20260928 12:30:03.185153 28050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c6bf06d793348dcbc915dbded9ec5af. Latest consensus state: current_term: 1 leader_uuid: "1c6bf06d793348dcbc915dbded9ec5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6bf06d793348dcbc915dbded9ec5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } } }
I20260928 12:30:03.185281 28051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c6bf06d793348dcbc915dbded9ec5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6bf06d793348dcbc915dbded9ec5af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36839 } } }
I20260928 12:30:03.185312 28050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:03.185377 28051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:03.185858 28058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:03.186033 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 27978
I20260928 12:30:03.186164 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:03.186372 7245 external_mini_cluster.cc:1428] Setting key 29ee72910ea6019a4737a328bb39a520
I20260928 12:30:03.186434 28058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:03.188421 28058 catalog_manager.cc:1347] Generated new cluster ID: 137de93decb84bf8af5d8067b0e9169f
I20260928 12:30:03.188472 28058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:03.193081 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:03.186846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43815 (local address 127.0.0.1:36839)
0928 12:30:03.187045 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:03.187055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:03.187287 (+ 232us) server_negotiation.cc:407] Connection header received
0928 12:30:03.187398 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:03.187406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:03.187437 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:03.187604 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:03.188270 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:03.188273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:03.188274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:03.188279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:03.188322 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:03.188336 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:03.192775 (+ 4439us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:03.192868 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 12:30:03.192911 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260928 12:30:03.193939 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:03.197549 28058 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:03.197701 28058 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:03.198469 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:03.197347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34919 (local address 127.0.0.1:36839)
0928 12:30:03.197510 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:03.197512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:03.197521 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:03.197582 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:03.197583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:03.197590 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:03.197652 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:03.198243 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:03.198244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:03.198244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:03.198248 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:03.198263 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:03.198268 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:03.198327 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:03.198399 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:30:03.198412 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:30:03.203809 28058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c6bf06d793348dcbc915dbded9ec5af: Generated new TSK 0
I20260928 12:30:03.203987 28058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:03.209936 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 27978
2026-09-28T12:30:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:03 dist-test-slave-cb7x krb5kdc[28077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:03 dist-test-slave-cb7x krb5kdc[28077](info): set up 2 sockets
Sep 28 12:30:03 dist-test-slave-cb7x krb5kdc[28077](info): commencing operation
krb5kdc: starting...
W20260928 12:30:04.248917 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:04 dist-test-slave-cb7x krb5kdc[28077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:04.364329 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:04.443228 28090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:04.443463 28090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:04.443500 28090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:04.444732 28090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:04.444803 28090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:04.444828 28090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:04.444856 28090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:04.444886 28090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:04.446318 28090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:04.446571 28090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:04.446810 28090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:04.451709 28098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:04.451704 28096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:04.451704 28095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:04.451843 28090 server_base.cc:1034] running on GCE node
I20260928 12:30:04.452174 28090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:04.452412 28090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:04.453557 28090 hybrid_clock.cc:648] HybridClock initialized: now 1790598604453508 us; error 61 us; skew 500 ppm
Sep 28 12:30:04 dist-test-slave-cb7x krb5kdc[28077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:04.456015 28090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:04.456460 28090 webserver.cc:460] Webserver started at http://127.0.0.1:38203/ using document root <none> and password file <none>
I20260928 12:30:04.456661 28090 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:04.456764 28090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:04.456854 28090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:04.457818 28090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "6eb335aab2dc42fbabca0fa76c24b23b"
format_stamp: "Formatted at 2026-09-28 12:30:04 on dist-test-slave-cb7x"
server_key: "6579efbc2c928a7fa17e898f5bfd173b"
server_key_iv: "68aefd7ade01ff60b14ca859468849d0"
server_key_version: "encryptionkey@0"
I20260928 12:30:04.458113 28090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "6eb335aab2dc42fbabca0fa76c24b23b"
format_stamp: "Formatted at 2026-09-28 12:30:04 on dist-test-slave-cb7x"
server_key: "6579efbc2c928a7fa17e898f5bfd173b"
server_key_iv: "68aefd7ade01ff60b14ca859468849d0"
server_key_version: "encryptionkey@0"
I20260928 12:30:04.459398 28090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:04.460155 28105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:04.460337 28090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:04.460422 28090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "6eb335aab2dc42fbabca0fa76c24b23b"
format_stamp: "Formatted at 2026-09-28 12:30:04 on dist-test-slave-cb7x"
server_key: "6579efbc2c928a7fa17e898f5bfd173b"
server_key_iv: "68aefd7ade01ff60b14ca859468849d0"
server_key_version: "encryptionkey@0"
I20260928 12:30:04.460475 28090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:04.470746 28090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:04.473045 28090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:04.473153 28090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:04.477088 28090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45721
I20260928 12:30:04.477177 28157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45721 every 8 connection(s)
I20260928 12:30:04.477634 28090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:04.478633 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28090
I20260928 12:30:04.478673 28158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:04.478747 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:04.478935 7245 external_mini_cluster.cc:1428] Setting key 4f53c59606b8a0558b54a3a571d73d11
I20260928 12:30:04.482065 28158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b: Bootstrap starting.
I20260928 12:30:04.482756 28158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:04.483037 28158 log.cc:824] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:04.483738 28158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b: No bootstrap required, opened a new log
Sep 28 12:30:04 dist-test-slave-cb7x krb5kdc[28077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:30:04.486418 28158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 12:30:04.486575 28158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:04.486620 28158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eb335aab2dc42fbabca0fa76c24b23b, State: Initialized, Role: FOLLOWER
I20260928 12:30:04.486753 28158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 12:30:04.486850 28158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:04.486893 28158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:04.486934 28158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:04.487453 28158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 12:30:04.487563 28158 leader_election.cc:302] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eb335aab2dc42fbabca0fa76c24b23b; no voters:
I20260928 12:30:04.487737 28158 leader_election.cc:288] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:04.487773 28163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:04.487937 28163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [term 1 LEADER]: Becoming Leader. State: Replica: 6eb335aab2dc42fbabca0fa76c24b23b, State: Running, Role: LEADER
I20260928 12:30:04.487998 28158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:04.488054 28163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } }
I20260928 12:30:04.488487 28164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } } }
I20260928 12:30:04.488565 28164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:04.488806 28165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eb335aab2dc42fbabca0fa76c24b23b. Latest consensus state: current_term: 1 leader_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eb335aab2dc42fbabca0fa76c24b23b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45721 } } }
I20260928 12:30:04.488931 28165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:04.489080 28170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:04.489598 28170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:04.490688 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:04.479504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:58689 (local address 127.0.0.1:45721)
0928 12:30:04.479733 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:04.479743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:04.479776 (+ 33us) server_negotiation.cc:407] Connection header received
0928 12:30:04.480315 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:04.480324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:04.480392 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:04.480556 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:04.481185 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:04.482531 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:04.483337 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:04.483804 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:04.483929 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:04.486530 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:04.486534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:04.486534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:04.486546 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:04.488790 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:04.489283 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:04.489286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:04.489287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:04.489402 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:04.489675 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:04.489676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:04.489677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:04.489972 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:04.490137 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:04.490275 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 12:30:04.490481 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260928 12:30:04.491545 28170 catalog_manager.cc:1347] Generated new cluster ID: b63e09c5aed9494195f550d477360bfc
I20260928 12:30:04.491663 28170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:04.492367 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:04.500926 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:04.493960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:56629 (local address 127.0.0.1:45721)
0928 12:30:04.493993 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:04.493994 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:04.494080 (+ 86us) server_negotiation.cc:407] Connection header received
0928 12:30:04.494171 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:04.494171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:04.494180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:04.494205 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:04.494803 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:04.495515 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:04.496490 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:04.496821 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:04.496856 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:04.498265 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:04.498266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:04.498267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:04.498272 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:04.499743 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:04.500105 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:04.500106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:04.500106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:04.500171 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:04.500370 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:04.500371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:04.500371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:04.500472 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:04.500535 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:04.500785 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 12:30:04.500842 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:30:04.524017 28170 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:04.524538 28170 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:04.530153 28170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eb335aab2dc42fbabca0fa76c24b23b: Generated new TSK 0
I20260928 12:30:04.530361 28170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:04.536260 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28090
W20260928 12:30:04.540660 28075 connection.cc:535] client connection to 127.0.0.1:45721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45721): Connection reset by peer (error 104)
2026-09-28T12:30:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:04 dist-test-slave-cb7x krb5kdc[28189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:04 dist-test-slave-cb7x krb5kdc[28189](info): set up 2 sockets
Sep 28 12:30:04 dist-test-slave-cb7x krb5kdc[28189](info): commencing operation
krb5kdc: starting...
W20260928 12:30:05.576741 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:05 dist-test-slave-cb7x krb5kdc[28189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:05.696851 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:05.776844 28202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:05.777048 28202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:05.777070 28202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:05.778244 28202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:05.778290 28202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:05.778304 28202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:05.778317 28202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:05.778329 28202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:05.779780 28202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:05.780038 28202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:05.780277 28202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:05.785125 28210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:05.785163 28208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:05.785140 28207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:05.785461 28202 server_base.cc:1034] running on GCE node
I20260928 12:30:05.785698 28202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:05.785944 28202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:05.787092 28202 hybrid_clock.cc:648] HybridClock initialized: now 1790598605787081 us; error 31 us; skew 500 ppm
Sep 28 12:30:05 dist-test-slave-cb7x krb5kdc[28189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:05.789760 28202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:05.790259 28202 webserver.cc:460] Webserver started at http://127.0.0.1:38097/ using document root <none> and password file <none>
I20260928 12:30:05.790496 28202 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:05.790566 28202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:05.790750 28202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:05.791867 28202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "67a349982faf4f25a6e1a30bae10c7fd"
format_stamp: "Formatted at 2026-09-28 12:30:05 on dist-test-slave-cb7x"
server_key: "b204141834644bbfe015867c7a9f4916"
server_key_iv: "d5cd821efaf192e81fbfca0ab8864ba8"
server_key_version: "encryptionkey@0"
I20260928 12:30:05.792236 28202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "67a349982faf4f25a6e1a30bae10c7fd"
format_stamp: "Formatted at 2026-09-28 12:30:05 on dist-test-slave-cb7x"
server_key: "b204141834644bbfe015867c7a9f4916"
server_key_iv: "d5cd821efaf192e81fbfca0ab8864ba8"
server_key_version: "encryptionkey@0"
I20260928 12:30:05.793699 28202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:05.794524 28217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:05.794785 28202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 12:30:05.794888 28202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "67a349982faf4f25a6e1a30bae10c7fd"
format_stamp: "Formatted at 2026-09-28 12:30:05 on dist-test-slave-cb7x"
server_key: "b204141834644bbfe015867c7a9f4916"
server_key_iv: "d5cd821efaf192e81fbfca0ab8864ba8"
server_key_version: "encryptionkey@0"
I20260928 12:30:05.794974 28202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:05.813419 28202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:05.815455 28202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:05.815624 28202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:05.819705 28202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40973
I20260928 12:30:05.819722 28269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40973 every 8 connection(s)
I20260928 12:30:05.820147 28202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:05.821189 28270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:05.821648 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28202
I20260928 12:30:05.821746 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:05.821955 7245 external_mini_cluster.cc:1428] Setting key 982e3e321e4e6195ca3fac5650b5633c
I20260928 12:30:05.824297 28270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd: Bootstrap starting.
I20260928 12:30:05.824991 28270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:05.825330 28270 log.cc:824] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:05.826014 28270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd: No bootstrap required, opened a new log
I20260928 12:30:05.828219 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:05.822481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:52747 (local address 127.0.0.1:40973)
0928 12:30:05.822683 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:05.822692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:05.822726 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:30:05.823250 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:05.823258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:05.823320 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:05.823481 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:05.823648 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:05.825012 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:05.826076 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:05.826666 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:05.826777 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:05.827415 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:05.827418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:05.827419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:05.827617 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:05.827672 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:05.827686 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:05.827787 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:05.827942 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 12:30:05.827975 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260928 12:30:05.828399 28270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a349982faf4f25a6e1a30bae10c7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 12:30:05.828547 28270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:05.828580 28270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a349982faf4f25a6e1a30bae10c7fd, State: Initialized, Role: FOLLOWER
I20260928 12:30:05.828718 28270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a349982faf4f25a6e1a30bae10c7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 12:30:05.828805 28270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:05.828899 28270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:05.828953 28270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:05.829284 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:05.829607 28270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a349982faf4f25a6e1a30bae10c7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 12:30:05.829725 28270 leader_election.cc:302] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a349982faf4f25a6e1a30bae10c7fd; no voters:
I20260928 12:30:05.829907 28270 leader_election.cc:288] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:05.829947 28275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:05.830127 28275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [term 1 LEADER]: Becoming Leader. State: Replica: 67a349982faf4f25a6e1a30bae10c7fd, State: Running, Role: LEADER
I20260928 12:30:05.830181 28270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:05.830297 28275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a349982faf4f25a6e1a30bae10c7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } }
I20260928 12:30:05.830627 28275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a349982faf4f25a6e1a30bae10c7fd. Latest consensus state: current_term: 1 leader_uuid: "67a349982faf4f25a6e1a30bae10c7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a349982faf4f25a6e1a30bae10c7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260928 12:30:05.830648 28277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a349982faf4f25a6e1a30bae10c7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a349982faf4f25a6e1a30bae10c7fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40973 } } }
I20260928 12:30:05.830739 28275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:05.830765 28277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:05.830977 28286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:05.831488 28286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:05.833137 28286 catalog_manager.cc:1347] Generated new cluster ID: f084c36b4a9c42edab2562d9d0653cde
I20260928 12:30:05.833185 28286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:05.835471 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:05.831868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:33407 (local address 127.0.0.1:40973)
0928 12:30:05.831900 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:05.831902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:05.832067 (+ 165us) server_negotiation.cc:407] Connection header received
0928 12:30:05.832119 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:05.832120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:05.832130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:05.832195 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:05.832354 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:05.833081 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:05.834092 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:05.834377 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:05.834418 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:05.835203 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:05.835205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:05.835205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:05.835209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:05.835222 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:05.835226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:05.835258 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:05.835393 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 12:30:05.835403 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:30:05.861220 28286 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:05.861781 28286 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:05.867046 28286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a349982faf4f25a6e1a30bae10c7fd: Generated new TSK 0
I20260928 12:30:05.867285 28286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:05.878700 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28202
W20260928 12:30:05.882871 28187 connection.cc:535] client connection to 127.0.0.1:40973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40973): Connection reset by peer (error 104)
2026-09-28T12:30:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:05 dist-test-slave-cb7x krb5kdc[28301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:05 dist-test-slave-cb7x krb5kdc[28301](info): set up 2 sockets
Sep 28 12:30:05 dist-test-slave-cb7x krb5kdc[28301](info): commencing operation
krb5kdc: starting...
W20260928 12:30:06.912835 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:06 dist-test-slave-cb7x krb5kdc[28301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:07.036582 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:07.117529 28314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:07.117739 28314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:07.117761 28314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:07.118903 28314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:07.118947 28314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:07.118960 28314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:07.118973 28314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:07.118984 28314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:07.120446 28314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:07.120682 28314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:07.120998 28314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:07.125722 28319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:07.125771 28320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:07.125823 28322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:07.125988 28314 server_base.cc:1034] running on GCE node
I20260928 12:30:07.126238 28314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:07.126528 28314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:07.127707 28314 hybrid_clock.cc:648] HybridClock initialized: now 1790598607127675 us; error 50 us; skew 500 ppm
Sep 28 12:30:07 dist-test-slave-cb7x krb5kdc[28301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:07.130288 28314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:07.130726 28314 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
I20260928 12:30:07.130996 28314 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:07.131043 28314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:07.131150 28314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:07.132150 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "e1afb4385b8e40a7ac045d48c8fb83fc"
format_stamp: "Formatted at 2026-09-28 12:30:07 on dist-test-slave-cb7x"
server_key: "c344bb6cdef1053c70ae35d2cef069f8"
server_key_iv: "b624ce70c99f103737bbe794f51f4910"
server_key_version: "encryptionkey@0"
I20260928 12:30:07.132454 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "e1afb4385b8e40a7ac045d48c8fb83fc"
format_stamp: "Formatted at 2026-09-28 12:30:07 on dist-test-slave-cb7x"
server_key: "c344bb6cdef1053c70ae35d2cef069f8"
server_key_iv: "b624ce70c99f103737bbe794f51f4910"
server_key_version: "encryptionkey@0"
I20260928 12:30:07.133839 28314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:07.134601 28329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:07.134781 28314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:07.134872 28314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "e1afb4385b8e40a7ac045d48c8fb83fc"
format_stamp: "Formatted at 2026-09-28 12:30:07 on dist-test-slave-cb7x"
server_key: "c344bb6cdef1053c70ae35d2cef069f8"
server_key_iv: "b624ce70c99f103737bbe794f51f4910"
server_key_version: "encryptionkey@0"
I20260928 12:30:07.134928 28314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:07.138198 28314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:07.140439 28314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:07.140543 28314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:07.144169 28314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44741
I20260928 12:30:07.144223 28381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44741 every 8 connection(s)
I20260928 12:30:07.144521 28314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:07.145526 28382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:07.147742 28382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc: Bootstrap starting.
I20260928 12:30:07.148317 28382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:07.148598 28382 log.cc:824] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:07.149336 28382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc: No bootstrap required, opened a new log
I20260928 12:30:07.150892 28382 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 12:30:07.151058 28382 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:07.151100 28382 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1afb4385b8e40a7ac045d48c8fb83fc, State: Initialized, Role: FOLLOWER
I20260928 12:30:07.151194 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28314
I20260928 12:30:07.151254 28382 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 12:30:07.151317 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:07.151350 28382 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:07.151386 28382 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:07.151438 28382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:07.151515 7245 external_mini_cluster.cc:1428] Setting key e96e9146f4db2f165a841ff8e4da43d2
I20260928 12:30:07.152129 28382 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 12:30:07.152243 28382 leader_election.cc:302] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1afb4385b8e40a7ac045d48c8fb83fc; no voters:
I20260928 12:30:07.152422 28382 leader_election.cc:288] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:07.152447 28387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:07.152547 28387 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [term 1 LEADER]: Becoming Leader. State: Replica: e1afb4385b8e40a7ac045d48c8fb83fc, State: Running, Role: LEADER
I20260928 12:30:07.152622 28387 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } }
I20260928 12:30:07.152702 28382 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:07.152956 28387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } } }
I20260928 12:30:07.153029 28387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:07.153151 28387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1afb4385b8e40a7ac045d48c8fb83fc. Latest consensus state: current_term: 1 leader_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1afb4385b8e40a7ac045d48c8fb83fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44741 } } }
I20260928 12:30:07.153194 28387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:07.153309 28391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:07.153954 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:07.152017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:43921 (local address 127.0.0.1:44741)
0928 12:30:07.152214 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:07.152225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:07.152259 (+ 34us) server_negotiation.cc:407] Connection header received
0928 12:30:07.152381 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:07.152386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:07.152409 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:07.152592 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:07.153444 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:07.153448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:07.153448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:07.153452 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:07.153498 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:07.153514 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:07.153579 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:07.153669 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:30:07.153721 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260928 12:30:07.154109 28391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:07.154762 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:07.155754 28391 catalog_manager.cc:1347] Generated new cluster ID: eee0adfd5002456b80671834cb918a7c
I20260928 12:30:07.155802 28391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:07.161787 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:07.156947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:44851 (local address 127.0.0.1:44741)
0928 12:30:07.156982 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:07.156984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:07.160827 (+ 3843us) server_negotiation.cc:407] Connection header received
0928 12:30:07.160882 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:07.160883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:07.160889 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:07.160958 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:07.161559 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:07.161560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:07.161560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:07.161564 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:07.161580 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:07.161584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:07.161656 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:07.161714 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:30:07.161727 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:07.179124 28391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:07.179324 28391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:07.188254 28391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1afb4385b8e40a7ac045d48c8fb83fc: Generated new TSK 0
I20260928 12:30:07.188445 28391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:07.194926 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28314
2026-09-28T12:30:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:07 dist-test-slave-cb7x krb5kdc[28413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:07 dist-test-slave-cb7x krb5kdc[28413](info): set up 2 sockets
Sep 28 12:30:07 dist-test-slave-cb7x krb5kdc[28413](info): commencing operation
krb5kdc: starting...
W20260928 12:30:08.240831 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:08 dist-test-slave-cb7x krb5kdc[28413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:08.359537 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:08.456414 28426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:08.456604 28426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:08.456626 28426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:08.457779 28426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:08.457823 28426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:08.457836 28426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:08.457849 28426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:08.457861 28426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:08.459215 28426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:08.459383 28426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:08.459542 28426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:08.464038 28434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:08.464048 28432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:08.464048 28431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:08.464075 28426 server_base.cc:1034] running on GCE node
I20260928 12:30:08.464551 28426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:08.464888 28426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:08.466032 28426 hybrid_clock.cc:648] HybridClock initialized: now 1790598608466015 us; error 34 us; skew 500 ppm
Sep 28 12:30:08 dist-test-slave-cb7x krb5kdc[28413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:08.468356 28426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:08.468860 28426 webserver.cc:460] Webserver started at http://127.0.0.1:39451/ using document root <none> and password file <none>
I20260928 12:30:08.469100 28426 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:08.469168 28426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:08.469307 28426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:08.470304 28426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "23c275f1992b4d3fa924015d578d3d95"
format_stamp: "Formatted at 2026-09-28 12:30:08 on dist-test-slave-cb7x"
server_key: "a9c992b2dd93538a5eb10e9730e786f9"
server_key_iv: "b38444ded7a3283c0b81ab07cb3e59cd"
server_key_version: "encryptionkey@0"
I20260928 12:30:08.470661 28426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "23c275f1992b4d3fa924015d578d3d95"
format_stamp: "Formatted at 2026-09-28 12:30:08 on dist-test-slave-cb7x"
server_key: "a9c992b2dd93538a5eb10e9730e786f9"
server_key_iv: "b38444ded7a3283c0b81ab07cb3e59cd"
server_key_version: "encryptionkey@0"
I20260928 12:30:08.471927 28426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:08.472625 28441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:08.472813 28426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:30:08.472910 28426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "23c275f1992b4d3fa924015d578d3d95"
format_stamp: "Formatted at 2026-09-28 12:30:08 on dist-test-slave-cb7x"
server_key: "a9c992b2dd93538a5eb10e9730e786f9"
server_key_iv: "b38444ded7a3283c0b81ab07cb3e59cd"
server_key_version: "encryptionkey@0"
I20260928 12:30:08.472996 28426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:08.493937 28426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:08.495759 28426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:08.495916 28426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:08.500011 28426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33505
I20260928 12:30:08.500047 28493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33505 every 8 connection(s)
I20260928 12:30:08.500411 28426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:08.501458 28494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:08.503676 28494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95: Bootstrap starting.
I20260928 12:30:08.504253 28494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:08.504264 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28426
I20260928 12:30:08.504385 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:08.504566 28494 log.cc:824] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:08.504585 7245 external_mini_cluster.cc:1428] Setting key 83e3b898f7b979a0749b24bd1acdacd3
I20260928 12:30:08.505381 28494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95: No bootstrap required, opened a new log
I20260928 12:30:08.507874 28494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c275f1992b4d3fa924015d578d3d95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260928 12:30:08.508033 28494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:08.508062 28494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c275f1992b4d3fa924015d578d3d95, State: Initialized, Role: FOLLOWER
I20260928 12:30:08.508193 28494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c275f1992b4d3fa924015d578d3d95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260928 12:30:08.508291 28494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:08.508327 28494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:08.508375 28494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:08.508977 28494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c275f1992b4d3fa924015d578d3d95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260928 12:30:08.509079 28494 leader_election.cc:302] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c275f1992b4d3fa924015d578d3d95; no voters:
I20260928 12:30:08.509227 28494 leader_election.cc:288] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:08.509301 28499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:08.509449 28494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:08.509470 28499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [term 1 LEADER]: Becoming Leader. State: Replica: 23c275f1992b4d3fa924015d578d3d95, State: Running, Role: LEADER
I20260928 12:30:08.509554 28499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c275f1992b4d3fa924015d578d3d95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260928 12:30:08.509950 28499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c275f1992b4d3fa924015d578d3d95. Latest consensus state: current_term: 1 leader_uuid: "23c275f1992b4d3fa924015d578d3d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c275f1992b4d3fa924015d578d3d95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } } }
I20260928 12:30:08.510041 28500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c275f1992b4d3fa924015d578d3d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c275f1992b4d3fa924015d578d3d95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } } }
I20260928 12:30:08.510082 28499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:08.510099 28500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:08.510416 28508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:08.510944 28508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 12:30:08 dist-test-slave-cb7x krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 12:30:08.516103 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:08.505104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:52953 (local address 127.0.0.1:33505)
0928 12:30:08.505293 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:08.505302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:08.505334 (+ 32us) server_negotiation.cc:407] Connection header received
0928 12:30:08.505862 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:08.505871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:08.505935 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:08.506077 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:08.506657 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:08.508080 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:08.509005 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:08.509625 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:08.509744 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:08.512576 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:08.512579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:08.512580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:08.512591 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:08.514485 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:08.514915 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:08.514918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:08.514919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:08.515023 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:08.515240 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:08.515241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:08.515241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:08.515538 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:08.515719 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:08.515801 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:30:08.515897 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 12:30:08.517189 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:08.517220 28508 catalog_manager.cc:1347] Generated new cluster ID: 982f07bb6c5143c583c469eca722d5cb
I20260928 12:30:08.517261 28508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:08.526036 28508 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:08.526506 28508 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:08.529527 28508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c275f1992b4d3fa924015d578d3d95: Generated new TSK 0
I20260928 12:30:08.529699 28508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:08.536190 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:08.518688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:48667 (local address 127.0.0.1:33505)
0928 12:30:08.518721 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:08.518723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:08.525769 (+ 7046us) server_negotiation.cc:407] Connection header received
0928 12:30:08.526006 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:08.526007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:08.526018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:08.526074 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:08.530146 (+ 4072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:08.530900 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:08.531587 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:08.532318 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:08.532393 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:08.533595 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:08.533597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:08.533597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:08.533604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:08.535084 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:08.535437 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:08.535439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:08.535439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:08.535497 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:08.535709 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:08.535710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:08.535711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:08.535834 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:08.535950 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:08.536040 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 12:30:08.536104 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:30:08.538131 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28426
W20260928 12:30:08.542004 28411 connection.cc:535] client connection to 127.0.0.1:33505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33505): Connection reset by peer (error 104)
2026-09-28T12:30:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:08 dist-test-slave-cb7x krb5kdc[28525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:08 dist-test-slave-cb7x krb5kdc[28525](info): set up 2 sockets
Sep 28 12:30:08 dist-test-slave-cb7x krb5kdc[28525](info): commencing operation
krb5kdc: starting...
W20260928 12:30:09.572839 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:09 dist-test-slave-cb7x krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:09.692355 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:09.772042 28538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:09.772267 28538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:09.772290 28538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:09.773473 28538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:09.773516 28538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:09.773530 28538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:09.773543 28538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:09.773555 28538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:09.775013 28538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:09.775267 28538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:09.775499 28538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:09.780424 28543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:09.780424 28546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:09.780457 28544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:09.780468 28538 server_base.cc:1034] running on GCE node
I20260928 12:30:09.780859 28538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:09.781177 28538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:09.782393 28538 hybrid_clock.cc:648] HybridClock initialized: now 1790598609782368 us; error 40 us; skew 500 ppm
Sep 28 12:30:09 dist-test-slave-cb7x krb5kdc[28525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:09.784842 28538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:09.785341 28538 webserver.cc:460] Webserver started at http://127.0.0.1:44747/ using document root <none> and password file <none>
I20260928 12:30:09.785568 28538 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:09.785640 28538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:09.785773 28538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:09.786852 28538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "703bde20015f42a68e8b21ebf67c4e7d"
format_stamp: "Formatted at 2026-09-28 12:30:09 on dist-test-slave-cb7x"
server_key: "e3055754823a9b0ecae3fbfded058772"
server_key_iv: "b98a6aff438df3edaba1937ed28c6929"
server_key_version: "encryptionkey@0"
I20260928 12:30:09.787230 28538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "703bde20015f42a68e8b21ebf67c4e7d"
format_stamp: "Formatted at 2026-09-28 12:30:09 on dist-test-slave-cb7x"
server_key: "e3055754823a9b0ecae3fbfded058772"
server_key_iv: "b98a6aff438df3edaba1937ed28c6929"
server_key_version: "encryptionkey@0"
I20260928 12:30:09.788597 28538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 12:30:09.789429 28553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:09.789602 28538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:30:09.789710 28538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "703bde20015f42a68e8b21ebf67c4e7d"
format_stamp: "Formatted at 2026-09-28 12:30:09 on dist-test-slave-cb7x"
server_key: "e3055754823a9b0ecae3fbfded058772"
server_key_iv: "b98a6aff438df3edaba1937ed28c6929"
server_key_version: "encryptionkey@0"
I20260928 12:30:09.789826 28538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:09.798681 28538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:09.800470 28538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:09.800627 28538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:09.804652 28538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46657
I20260928 12:30:09.804733 28605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46657 every 8 connection(s)
I20260928 12:30:09.805193 28538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:09.806068 28606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:09.806998 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28538
I20260928 12:30:09.807124 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:09.807346 7245 external_mini_cluster.cc:1428] Setting key c92f7d7ea810b124e0c9d1d7c72fad58
I20260928 12:30:09.809388 28606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d: Bootstrap starting.
I20260928 12:30:09.810034 28606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:09.810352 28606 log.cc:824] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:09.811118 28606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d: No bootstrap required, opened a new log
I20260928 12:30:09.812925 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:09.807902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:34531 (local address 127.0.0.1:46657)
0928 12:30:09.808076 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:09.808085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:09.808120 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:30:09.808512 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:09.808518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:09.808565 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:09.808745 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:09.808936 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:09.809889 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:09.810970 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:09.811427 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:09.811515 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:09.812242 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:09.812245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:09.812245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:09.812401 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:09.812430 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:09.812440 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:09.812507 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:09.812709 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 12:30:09.812742 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 12:30:09.813625 28606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703bde20015f42a68e8b21ebf67c4e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260928 12:30:09.813799 28606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:09.813836 28606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703bde20015f42a68e8b21ebf67c4e7d, State: Initialized, Role: FOLLOWER
I20260928 12:30:09.813984 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:09.813961 28606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703bde20015f42a68e8b21ebf67c4e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260928 12:30:09.814064 28606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:09.814101 28606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:09.814152 28606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:09.814682 28606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703bde20015f42a68e8b21ebf67c4e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260928 12:30:09.814800 28606 leader_election.cc:302] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 703bde20015f42a68e8b21ebf67c4e7d; no voters:
I20260928 12:30:09.814981 28606 leader_election.cc:288] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:09.815069 28613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:09.815248 28606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:09.815541 28613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [term 1 LEADER]: Becoming Leader. State: Replica: 703bde20015f42a68e8b21ebf67c4e7d, State: Running, Role: LEADER
I20260928 12:30:09.815722 28613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703bde20015f42a68e8b21ebf67c4e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260928 12:30:09.816299 28616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "703bde20015f42a68e8b21ebf67c4e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703bde20015f42a68e8b21ebf67c4e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } } }
I20260928 12:30:09.816378 28616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:09.816851 28626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:09.817003 28618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 703bde20015f42a68e8b21ebf67c4e7d. Latest consensus state: current_term: 1 leader_uuid: "703bde20015f42a68e8b21ebf67c4e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703bde20015f42a68e8b21ebf67c4e7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } } }
I20260928 12:30:09.817093 28618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:09.817626 28626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:09.818992 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:09.815393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:44963 (local address 127.0.0.1:46657)
0928 12:30:09.815445 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:09.815447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:09.815591 (+ 144us) server_negotiation.cc:407] Connection header received
0928 12:30:09.815683 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:09.815684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:09.815693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:09.815720 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:09.815910 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:09.816636 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:09.817599 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:09.817888 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:09.817975 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:09.818744 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:09.818745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:09.818745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:09.818748 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:09.818760 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:09.818763 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:09.818840 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:09.818909 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 12:30:09.818918 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:30:09.825162 28626 catalog_manager.cc:1347] Generated new cluster ID: 80c5f30d22cd409893d5a4d047cd2f86
I20260928 12:30:09.825207 28626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:09.832387 28626 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:09.833096 28626 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:09.844235 28626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 703bde20015f42a68e8b21ebf67c4e7d: Generated new TSK 0
I20260928 12:30:09.844439 28626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:09.851310 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28538
W20260928 12:30:09.855499 28523 connection.cc:535] client connection to 127.0.0.1:46657 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46657): Connection reset by peer (error 104)
2026-09-28T12:30:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:09 dist-test-slave-cb7x krb5kdc[28638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:09 dist-test-slave-cb7x krb5kdc[28638](info): set up 2 sockets
Sep 28 12:30:09 dist-test-slave-cb7x krb5kdc[28638](info): commencing operation
krb5kdc: starting...
W20260928 12:30:10.888942 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:10 dist-test-slave-cb7x krb5kdc[28638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 12:30:11.007874 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:11.087580 28651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:11.087783 28651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:11.087806 28651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:11.088927 28651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:11.088963 28651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:11.088976 28651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:11.088989 28651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:11.089000 28651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:11.090358 28651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:11.090546 28651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:11.090711 28651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:11.095531 28659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:11.095634 28651 server_base.cc:1034] running on GCE node
W20260928 12:30:11.095556 28656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:11.095530 28657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:11.096130 28651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:11.096431 28651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:11.097609 28651 hybrid_clock.cc:648] HybridClock initialized: now 1790598611097593 us; error 62 us; skew 500 ppm
Sep 28 12:30:11 dist-test-slave-cb7x krb5kdc[28638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:11.100024 28651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 12:30:11.100463 28651 webserver.cc:460] Webserver started at http://127.0.0.1:40517/ using document root <none> and password file <none>
I20260928 12:30:11.100742 28651 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:11.100790 28651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:11.100916 28651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:11.101933 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "860b59a89b654275b973fe7e1da7a50e"
format_stamp: "Formatted at 2026-09-28 12:30:11 on dist-test-slave-cb7x"
server_key: "bf7d8c6c20d7e0f6565cf7461e8099df"
server_key_iv: "539670a2d26b0619d5dc6ea322827bca"
server_key_version: "encryptionkey@0"
I20260928 12:30:11.102303 28651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "860b59a89b654275b973fe7e1da7a50e"
format_stamp: "Formatted at 2026-09-28 12:30:11 on dist-test-slave-cb7x"
server_key: "bf7d8c6c20d7e0f6565cf7461e8099df"
server_key_iv: "539670a2d26b0619d5dc6ea322827bca"
server_key_version: "encryptionkey@0"
I20260928 12:30:11.103613 28651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:11.104349 28666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:11.104535 28651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:30:11.104633 28651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "860b59a89b654275b973fe7e1da7a50e"
format_stamp: "Formatted at 2026-09-28 12:30:11 on dist-test-slave-cb7x"
server_key: "bf7d8c6c20d7e0f6565cf7461e8099df"
server_key_iv: "539670a2d26b0619d5dc6ea322827bca"
server_key_version: "encryptionkey@0"
I20260928 12:30:11.104744 28651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:11.108704 28651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:11.110500 28651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:11.110653 28651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:11.114579 28651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39297
I20260928 12:30:11.114611 28718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39297 every 8 connection(s)
I20260928 12:30:11.114960 28651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:11.115900 28719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:11.118211 28719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e: Bootstrap starting.
I20260928 12:30:11.118733 28719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:11.118957 28719 log.cc:824] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:11.119607 28719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e: No bootstrap required, opened a new log
I20260928 12:30:11.121168 28719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860b59a89b654275b973fe7e1da7a50e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260928 12:30:11.121300 28719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:11.121322 28719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860b59a89b654275b973fe7e1da7a50e, State: Initialized, Role: FOLLOWER
I20260928 12:30:11.121420 28719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860b59a89b654275b973fe7e1da7a50e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260928 12:30:11.121470 28719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:11.121490 28719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:11.121556 28719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:11.122097 28719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860b59a89b654275b973fe7e1da7a50e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260928 12:30:11.122186 28719 leader_election.cc:302] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 860b59a89b654275b973fe7e1da7a50e; no voters:
I20260928 12:30:11.122370 28719 leader_election.cc:288] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:11.122412 28722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:11.122506 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28651
I20260928 12:30:11.122619 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:11.122629 28722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [term 1 LEADER]: Becoming Leader. State: Replica: 860b59a89b654275b973fe7e1da7a50e, State: Running, Role: LEADER
I20260928 12:30:11.122644 28719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:11.122830 7245 external_mini_cluster.cc:1428] Setting key 9557a6460afdcadc7c76dd6c34aab3f5
I20260928 12:30:11.122822 28722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860b59a89b654275b973fe7e1da7a50e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } }
I20260928 12:30:11.123987 28724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "860b59a89b654275b973fe7e1da7a50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860b59a89b654275b973fe7e1da7a50e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } } }
I20260928 12:30:11.124009 28723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860b59a89b654275b973fe7e1da7a50e. Latest consensus state: current_term: 1 leader_uuid: "860b59a89b654275b973fe7e1da7a50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860b59a89b654275b973fe7e1da7a50e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39297 } } }
I20260928 12:30:11.124143 28723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:11.124763 28724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e [sys.catalog]: This master's current role is: LEADER
W20260928 12:30:11.125159 28739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:11.125236 28739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:11.125288 28740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:11.125331 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:11.123279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:37729 (local address 127.0.0.1:39297)
0928 12:30:11.123600 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:11.123608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:11.123690 (+ 82us) server_negotiation.cc:407] Connection header received
0928 12:30:11.123803 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:11.123810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:11.123842 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:11.124259 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:11.124798 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:11.124802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:11.124802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:11.124806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:11.124845 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:11.124873 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:11.124944 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:11.125066 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 12:30:11.125114 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":74,"threads_started":1}
I20260928 12:30:11.125432 28740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:11.126250 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:11.126885 28740 catalog_manager.cc:1347] Generated new cluster ID: b8847a69ef974580960b9b392d814adf
I20260928 12:30:11.126937 28740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:11.134078 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:11.127778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.175:49437 (local address 127.0.0.1:39297)
0928 12:30:11.127818 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:11.127820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:11.132969 (+ 5149us) server_negotiation.cc:407] Connection header received
0928 12:30:11.133061 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:11.133061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:11.133068 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:11.133119 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:11.133807 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:11.133809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:11.133809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:11.133814 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:11.133829 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:11.133834 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:11.133892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:11.133953 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 12:30:11.133970 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:11.143541 28740 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:11.143712 28740 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:11.155918 28740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 860b59a89b654275b973fe7e1da7a50e: Generated new TSK 0
I20260928 12:30:11.156123 28740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:11.161499 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28651
2026-09-28T12:30:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:11 dist-test-slave-cb7x krb5kdc[28750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:11 dist-test-slave-cb7x krb5kdc[28750](info): set up 2 sockets
Sep 28 12:30:11 dist-test-slave-cb7x krb5kdc[28750](info): commencing operation
krb5kdc: starting...
W20260928 12:30:12.192832 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:12 dist-test-slave-cb7x krb5kdc[28750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:12.312752 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37195
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:43829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:12.391048 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:12.391256 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:12.391286 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:12.392457 28763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:12.392501 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:12.392516 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:12.392558 28763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:12.392590 28763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:12.394042 28763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:43829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:12.394302 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:12.394538 28763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:12.399322 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:12.399336 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:12.399374 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:12.399492 28763 server_base.cc:1034] running on GCE node
I20260928 12:30:12.399850 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:12.400203 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:12.401363 28763 hybrid_clock.cc:648] HybridClock initialized: now 1790598612401356 us; error 57 us; skew 500 ppm
Sep 28 12:30:12 dist-test-slave-cb7x krb5kdc[28750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598612, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:12.403817 28763 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:12.404287 28763 webserver.cc:460] Webserver started at http://127.7.19.126:38273/ using document root <none> and password file <none>
I20260928 12:30:12.404479 28763 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:12.404520 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:12.404608 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:12.405696 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "5b77433ee86b4ec6a411593afb790415"
format_stamp: "Formatted at 2026-09-28 12:30:12 on dist-test-slave-cb7x"
server_key: "956076ef7d1249291933b25bc4c1a803"
server_key_iv: "11238ff997aef47920d1ea6e34a22cb9"
server_key_version: "encryptionkey@0"
I20260928 12:30:12.406013 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "5b77433ee86b4ec6a411593afb790415"
format_stamp: "Formatted at 2026-09-28 12:30:12 on dist-test-slave-cb7x"
server_key: "956076ef7d1249291933b25bc4c1a803"
server_key_iv: "11238ff997aef47920d1ea6e34a22cb9"
server_key_version: "encryptionkey@0"
I20260928 12:30:12.407347 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:12.408088 28778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:12.408308 28763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:12.408397 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "5b77433ee86b4ec6a411593afb790415"
format_stamp: "Formatted at 2026-09-28 12:30:12 on dist-test-slave-cb7x"
server_key: "956076ef7d1249291933b25bc4c1a803"
server_key_iv: "11238ff997aef47920d1ea6e34a22cb9"
server_key_version: "encryptionkey@0"
I20260928 12:30:12.408454 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:12.421257 28763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:12.423055 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:12.423179 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:12.427193 28763 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:37195
I20260928 12:30:12.427464 28830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:37195 every 8 connection(s)
I20260928 12:30:12.427579 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:12.428282 28831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:12.430606 28831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415: Bootstrap starting.
I20260928 12:30:12.431178 28831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:12.431463 28831 log.cc:824] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:12.432240 28831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415: No bootstrap required, opened a new log
I20260928 12:30:12.433947 28831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b77433ee86b4ec6a411593afb790415" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37195 } }
I20260928 12:30:12.434137 28831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:12.434191 28831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b77433ee86b4ec6a411593afb790415, State: Initialized, Role: FOLLOWER
I20260928 12:30:12.434333 28831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b77433ee86b4ec6a411593afb790415" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37195 } }
I20260928 12:30:12.434419 28831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:12.434458 28831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:12.434510 28831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:12.435374 28831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b77433ee86b4ec6a411593afb790415" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37195 } }
I20260928 12:30:12.435493 28831 leader_election.cc:302] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b77433ee86b4ec6a411593afb790415; no voters:
I20260928 12:30:12.435688 28831 leader_election.cc:288] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:12.435753 28834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:12.435950 28831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:12.435983 28834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [term 1 LEADER]: Becoming Leader. State: Replica: 5b77433ee86b4ec6a411593afb790415, State: Running, Role: LEADER
I20260928 12:30:12.436110 28834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b77433ee86b4ec6a411593afb790415" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37195 } }
I20260928 12:30:12.436518 28836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b77433ee86b4ec6a411593afb790415. Latest consensus state: current_term: 1 leader_uuid: "5b77433ee86b4ec6a411593afb790415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b77433ee86b4ec6a411593afb790415" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37195 } } }
I20260928 12:30:12.436604 28836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:12.436791 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b77433ee86b4ec6a411593afb790415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b77433ee86b4ec6a411593afb790415" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37195 } } }
I20260928 12:30:12.436857 28843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:12.437151 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:12.437170 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28763
I20260928 12:30:12.437283 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:12.437492 7245 external_mini_cluster.cc:1428] Setting key bf4a5cc55738630333199871eeeb8229
I20260928 12:30:12.437661 28843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:12.440135 28843 catalog_manager.cc:1347] Generated new cluster ID: 167782cd072d422fb9f0ec6bbbd8724c
I20260928 12:30:12.440186 28843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:12 dist-test-slave-cb7x krb5kdc[28750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:12.449513 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:12.437973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46021 (local address 127.7.19.126:37195)
0928 12:30:12.438520 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:12.438529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:12.438564 (+ 35us) server_negotiation.cc:407] Connection header received
0928 12:30:12.438678 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:12.438685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:12.438759 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:12.438899 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:12.439783 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:12.441168 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:12.442273 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:12.442867 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:12.442966 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:12.445804 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:12.445809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:12.445809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:12.445821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:12.447796 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:12.448219 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:12.448222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:12.448223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:12.448343 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:12.448578 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:12.448579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:12.448579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:12.448911 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:12.449065 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:12.449180 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 12:30:12.449282 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":76,"threads_started":1}
I20260928 12:30:12.450480 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:12.458997 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:12.451864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39375 (local address 127.7.19.126:37195)
0928 12:30:12.451896 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:12.451897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:12.452007 (+ 110us) server_negotiation.cc:407] Connection header received
0928 12:30:12.452061 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:12.452062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:12.452071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:12.452123 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:12.452741 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:12.453458 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:12.454467 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:12.454775 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:12.454836 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:12.456206 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:12.456207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:12.456207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:12.456213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:12.457796 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:12.458222 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:12.458223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:12.458223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:12.458275 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:12.458500 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:12.458501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:12.458501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:12.458608 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:12.458694 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:12.458855 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 12:30:12.458911 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:30:12.462625 28843 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:12.463353 28843 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:12.468225 28843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b77433ee86b4ec6a411593afb790415: Generated new TSK 0
I20260928 12:30:12.468405 28843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:12.471889 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28763
W20260928 12:30:12.476121 28748 connection.cc:535] client connection to 127.7.19.126:37195 recv error: Network error: failed to read from TLS socket (remote: 127.7.19.126:37195): Connection reset by peer (error 104)
2026-09-28T12:30:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:12 dist-test-slave-cb7x krb5kdc[28862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:12 dist-test-slave-cb7x krb5kdc[28862](info): set up 2 sockets
Sep 28 12:30:12 dist-test-slave-cb7x krb5kdc[28862](info): commencing operation
krb5kdc: starting...
W20260928 12:30:13.516995 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:13 dist-test-slave-cb7x krb5kdc[28862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:13.636118 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45705
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:13.715955 28875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:13.716171 28875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:13.716195 28875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:13.717402 28875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:13.717448 28875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:13.717464 28875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:13.717478 28875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:13.717489 28875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:13.719017 28875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:45705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:13.719285 28875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:13.719517 28875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:13.724496 28881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:13.724589 28875 server_base.cc:1034] running on GCE node
W20260928 12:30:13.724659 28880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:13.724735 28883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:13.725113 28875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:13.725421 28875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:13.726610 28875 hybrid_clock.cc:648] HybridClock initialized: now 1790598613726594 us; error 31 us; skew 500 ppm
Sep 28 12:30:13 dist-test-slave-cb7x krb5kdc[28862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598613, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:13.737183 28875 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:13.737692 28875 webserver.cc:460] Webserver started at http://127.7.19.126:45923/ using document root <none> and password file <none>
I20260928 12:30:13.737939 28875 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:13.738008 28875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:13.738164 28875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:13.739259 28875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "9c2068174ca746be8be7151689a0ced5"
format_stamp: "Formatted at 2026-09-28 12:30:13 on dist-test-slave-cb7x"
server_key: "2534ca8d295f511ff922a4c9dbb2b2c6"
server_key_iv: "81340364ed385819dd10b7596426f866"
server_key_version: "encryptionkey@0"
I20260928 12:30:13.739622 28875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "9c2068174ca746be8be7151689a0ced5"
format_stamp: "Formatted at 2026-09-28 12:30:13 on dist-test-slave-cb7x"
server_key: "2534ca8d295f511ff922a4c9dbb2b2c6"
server_key_iv: "81340364ed385819dd10b7596426f866"
server_key_version: "encryptionkey@0"
I20260928 12:30:13.741142 28875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:13.741890 28890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:13.742082 28875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:13.742184 28875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "9c2068174ca746be8be7151689a0ced5"
format_stamp: "Formatted at 2026-09-28 12:30:13 on dist-test-slave-cb7x"
server_key: "2534ca8d295f511ff922a4c9dbb2b2c6"
server_key_iv: "81340364ed385819dd10b7596426f866"
server_key_version: "encryptionkey@0"
I20260928 12:30:13.742266 28875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:13.764217 28875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:13.766595 28875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:13.766772 28875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:13.770653 28875 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:45705
I20260928 12:30:13.770696 28942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:45705 every 8 connection(s)
I20260928 12:30:13.771056 28875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:13.771888 28943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:13.774257 28943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5: Bootstrap starting.
I20260928 12:30:13.774834 28943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:13.775120 28943 log.cc:824] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:13.775815 28943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5: No bootstrap required, opened a new log
I20260928 12:30:13.777383 28943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2068174ca746be8be7151689a0ced5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45705 } }
I20260928 12:30:13.777518 28943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:13.777539 28943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2068174ca746be8be7151689a0ced5, State: Initialized, Role: FOLLOWER
I20260928 12:30:13.777738 28943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2068174ca746be8be7151689a0ced5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45705 } }
I20260928 12:30:13.777804 28943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:13.777827 28943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:13.777892 28943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:13.778694 28943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2068174ca746be8be7151689a0ced5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45705 } }
I20260928 12:30:13.778803 28943 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2068174ca746be8be7151689a0ced5; no voters:
I20260928 12:30:13.778959 28943 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:13.778995 28946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:13.779208 28943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:13.779304 28946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [term 1 LEADER]: Becoming Leader. State: Replica: 9c2068174ca746be8be7151689a0ced5, State: Running, Role: LEADER
I20260928 12:30:13.779434 28946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2068174ca746be8be7151689a0ced5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45705 } }
I20260928 12:30:13.779868 28948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2068174ca746be8be7151689a0ced5. Latest consensus state: current_term: 1 leader_uuid: "9c2068174ca746be8be7151689a0ced5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2068174ca746be8be7151689a0ced5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45705 } } }
I20260928 12:30:13.779876 28947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2068174ca746be8be7151689a0ced5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2068174ca746be8be7151689a0ced5" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 45705 } } }
I20260928 12:30:13.780023 28947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:13.780234 28948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:13.780382 28955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:13.780848 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28875
I20260928 12:30:13.780967 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:13.781148 7245 external_mini_cluster.cc:1428] Setting key 0f1ee0a703757b35d3088ee3f19898ec
I20260928 12:30:13.781613 28955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:13.783037 28955 catalog_manager.cc:1347] Generated new cluster ID: c5d085c892c14a6990adaafe87e0a559
I20260928 12:30:13.783099 28955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:13.790578 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:13.781596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41707 (local address 127.7.19.126:45705)
0928 12:30:13.785010 (+ 3414us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:13.785018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:13.785047 (+ 29us) server_negotiation.cc:407] Connection header received
0928 12:30:13.785152 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:13.785158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:13.785217 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:13.785334 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:13.785589 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:13.787302 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:13.788440 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:13.789053 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:13.789145 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:13.789880 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:13.789883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:13.789884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:13.790089 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:13.790123 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:13.790137 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:13.790221 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:13.790349 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 12:30:13.790374 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3359,"thread_start_us":64,"threads_started":1}
I20260928 12:30:13.791473 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:13.800316 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:13.792981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41243 (local address 127.7.19.126:45705)
0928 12:30:13.793024 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:13.793025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:13.796887 (+ 3862us) server_negotiation.cc:407] Connection header received
0928 12:30:13.796969 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:13.796970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:13.796980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:13.797064 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:13.797223 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:13.797976 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:13.799009 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:13.799334 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:13.799391 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:13.800093 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:13.800094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:13.800094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:13.800098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:13.800109 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:13.800112 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:13.800159 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:13.800236 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 12:30:13.800248 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:30:13.814332 28955 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:13.814908 28955 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:13.828205 28955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2068174ca746be8be7151689a0ced5: Generated new TSK 0
I20260928 12:30:13.828393 28955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:13.834244 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28875
W20260928 12:30:13.838537 28860 connection.cc:535] client connection to 127.7.19.126:45705 recv error: Network error: failed to read from TLS socket (remote: 127.7.19.126:45705): Connection reset by peer (error 104)
2026-09-28T12:30:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:13 dist-test-slave-cb7x krb5kdc[28974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:13 dist-test-slave-cb7x krb5kdc[28974](info): set up 2 sockets
Sep 28 12:30:13 dist-test-slave-cb7x krb5kdc[28974](info): commencing operation
krb5kdc: starting...
W20260928 12:30:14.873329 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:14 dist-test-slave-cb7x krb5kdc[28974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:14.989832 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38031
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:38031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:15.069594 28987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:15.069796 28987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:15.069819 28987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:15.070904 28987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:15.070940 28987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:15.070953 28987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:15.070966 28987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:15.070978 28987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:15.072335 28987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:38031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:15.072521 28987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:15.072713 28987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:15.077732 28993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:15.077732 28992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:15.077800 28995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:15.077768 28987 server_base.cc:1034] running on GCE node
I20260928 12:30:15.078116 28987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:15.078420 28987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:15.079581 28987 hybrid_clock.cc:648] HybridClock initialized: now 1790598615079567 us; error 31 us; skew 500 ppm
Sep 28 12:30:15 dist-test-slave-cb7x krb5kdc[28974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598615, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:15.082002 28987 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:15.082422 28987 webserver.cc:460] Webserver started at http://127.7.19.126:41833/ using document root <none> and password file <none>
I20260928 12:30:15.082634 28987 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:15.082700 28987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:15.082840 28987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:15.083904 28987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "0c9fbcafc6644666ac93352b757911b2"
format_stamp: "Formatted at 2026-09-28 12:30:15 on dist-test-slave-cb7x"
server_key: "05f1e939de5310a9f074ee5d0242cf63"
server_key_iv: "2ea8008e91d1d7b55f95161d609f6e52"
server_key_version: "encryptionkey@0"
I20260928 12:30:15.084259 28987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "0c9fbcafc6644666ac93352b757911b2"
format_stamp: "Formatted at 2026-09-28 12:30:15 on dist-test-slave-cb7x"
server_key: "05f1e939de5310a9f074ee5d0242cf63"
server_key_iv: "2ea8008e91d1d7b55f95161d609f6e52"
server_key_version: "encryptionkey@0"
I20260928 12:30:15.085657 28987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:15.086410 29002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:15.086578 28987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:30:15.086674 28987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "0c9fbcafc6644666ac93352b757911b2"
format_stamp: "Formatted at 2026-09-28 12:30:15 on dist-test-slave-cb7x"
server_key: "05f1e939de5310a9f074ee5d0242cf63"
server_key_iv: "2ea8008e91d1d7b55f95161d609f6e52"
server_key_version: "encryptionkey@0"
I20260928 12:30:15.086753 28987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:15.090080 28987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:15.091813 28987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:15.091958 28987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:15.095602 28987 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:38031
I20260928 12:30:15.095633 29054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:38031 every 8 connection(s)
I20260928 12:30:15.095925 28987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:15.096921 29055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:15.099004 29055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2: Bootstrap starting.
I20260928 12:30:15.099498 29055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:15.099725 29055 log.cc:824] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:15.100329 29055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2: No bootstrap required, opened a new log
I20260928 12:30:15.101914 29055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c9fbcafc6644666ac93352b757911b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38031 } }
I20260928 12:30:15.102042 29055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:15.102064 29055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9fbcafc6644666ac93352b757911b2, State: Initialized, Role: FOLLOWER
I20260928 12:30:15.102190 29055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c9fbcafc6644666ac93352b757911b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38031 } }
I20260928 12:30:15.102267 29055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:15.102288 29055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:15.102315 29055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:15.103053 29055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c9fbcafc6644666ac93352b757911b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38031 } }
I20260928 12:30:15.103132 29055 leader_election.cc:302] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c9fbcafc6644666ac93352b757911b2; no voters:
I20260928 12:30:15.103246 29055 leader_election.cc:288] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:15.103308 29058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:15.103475 29058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [term 1 LEADER]: Becoming Leader. State: Replica: 0c9fbcafc6644666ac93352b757911b2, State: Running, Role: LEADER
I20260928 12:30:15.103516 29055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:15.103565 29058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c9fbcafc6644666ac93352b757911b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38031 } }
I20260928 12:30:15.103955 29059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c9fbcafc6644666ac93352b757911b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c9fbcafc6644666ac93352b757911b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38031 } } }
I20260928 12:30:15.104048 29059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:15.103960 29060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c9fbcafc6644666ac93352b757911b2. Latest consensus state: current_term: 1 leader_uuid: "0c9fbcafc6644666ac93352b757911b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c9fbcafc6644666ac93352b757911b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 38031 } } }
I20260928 12:30:15.104138 29060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:15.104315 29067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:15.104386 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 28987
I20260928 12:30:15.104490 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:15.104743 7245 external_mini_cluster.cc:1428] Setting key 2fdbc313f4793a83da5ec4772868e549
I20260928 12:30:15.104863 29067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:15.106483 29067 catalog_manager.cc:1347] Generated new cluster ID: cb9bf174f71b42e58100e48dccac0164
I20260928 12:30:15.106550 29067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:15.107782 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:15.105338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55383 (local address 127.7.19.126:38031)
0928 12:30:15.106193 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:15.106201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:15.106229 (+ 28us) server_negotiation.cc:407] Connection header received
0928 12:30:15.106372 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:15.106392 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:15.106434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:15.106591 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:15.107279 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:15.107283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:15.107283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:15.107287 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:15.107329 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:15.107367 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:15.107458 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:15.107538 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 12:30:15.107658 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":92,"threads_started":1}
I20260928 12:30:15.108577 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:15.114187 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:15.113013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59901 (local address 127.7.19.126:38031)
0928 12:30:15.113046 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:15.113048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:15.113253 (+ 205us) server_negotiation.cc:407] Connection header received
0928 12:30:15.113324 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:15.113325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:15.113331 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:15.113380 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:15.113957 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:15.113958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:15.113958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:15.113963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:15.113975 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:15.113979 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:15.114035 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:15.114094 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 12:30:15.114109 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:30:15.117385 29067 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:15.117547 29067 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:15.127815 29067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c9fbcafc6644666ac93352b757911b2: Generated new TSK 0
I20260928 12:30:15.128018 29067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:15.133281 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 28987
2026-09-28T12:30:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:15 dist-test-slave-cb7x krb5kdc[29086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:15 dist-test-slave-cb7x krb5kdc[29086](info): set up 2 sockets
Sep 28 12:30:15 dist-test-slave-cb7x krb5kdc[29086](info): commencing operation
krb5kdc: starting...
W20260928 12:30:16.160903 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:16 dist-test-slave-cb7x krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:16.284992 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37915
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:16.365113 29099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:16.365352 29099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:16.365388 29099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:16.366544 29099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:16.366618 29099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:16.366642 29099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:16.366663 29099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:16.366688 29099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:16.368201 29099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:16.368472 29099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:16.368731 29099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:16.373587 29104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:16.373553 29107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:16.373587 29105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:16.373620 29099 server_base.cc:1034] running on GCE node
I20260928 12:30:16.374033 29099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:16.374372 29099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:16.375536 29099 hybrid_clock.cc:648] HybridClock initialized: now 1790598616375531 us; error 36 us; skew 500 ppm
Sep 28 12:30:16 dist-test-slave-cb7x krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598616, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:16.378047 29099 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:16.378517 29099 webserver.cc:460] Webserver started at http://127.7.19.126:42181/ using document root <none> and password file <none>
I20260928 12:30:16.378736 29099 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:16.378804 29099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:16.378953 29099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:16.380020 29099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "d9a390519aae4143a8ad47c4b6579722"
format_stamp: "Formatted at 2026-09-28 12:30:16 on dist-test-slave-cb7x"
server_key: "37a7b264a569e9d466d72cbbccf612b4"
server_key_iv: "d911c5e6178eb9ccd00b6542a632cba2"
server_key_version: "encryptionkey@0"
I20260928 12:30:16.380385 29099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "d9a390519aae4143a8ad47c4b6579722"
format_stamp: "Formatted at 2026-09-28 12:30:16 on dist-test-slave-cb7x"
server_key: "37a7b264a569e9d466d72cbbccf612b4"
server_key_iv: "d911c5e6178eb9ccd00b6542a632cba2"
server_key_version: "encryptionkey@0"
I20260928 12:30:16.381866 29099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:16.382627 29114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:16.382871 29099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:16.382980 29099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "d9a390519aae4143a8ad47c4b6579722"
format_stamp: "Formatted at 2026-09-28 12:30:16 on dist-test-slave-cb7x"
server_key: "37a7b264a569e9d466d72cbbccf612b4"
server_key_iv: "d911c5e6178eb9ccd00b6542a632cba2"
server_key_version: "encryptionkey@0"
I20260928 12:30:16.383065 29099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:16.404479 29099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:16.406915 29099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:16.407088 29099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:16.411360 29099 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:37915
I20260928 12:30:16.411413 29166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:37915 every 8 connection(s)
I20260928 12:30:16.411793 29099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:16.412662 29167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:16.414991 29167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722: Bootstrap starting.
I20260928 12:30:16.415575 29167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:16.415866 29167 log.cc:824] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:16.416766 29167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722: No bootstrap required, opened a new log
I20260928 12:30:16.418474 29167 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a390519aae4143a8ad47c4b6579722" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37915 } }
I20260928 12:30:16.418661 29167 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:16.418712 29167 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a390519aae4143a8ad47c4b6579722, State: Initialized, Role: FOLLOWER
I20260928 12:30:16.418864 29167 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a390519aae4143a8ad47c4b6579722" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37915 } }
I20260928 12:30:16.418948 29167 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:16.418988 29167 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:16.419042 29167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:16.419646 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29099
I20260928 12:30:16.419734 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:16.419932 7245 external_mini_cluster.cc:1428] Setting key 1d8d984e8f43c3fe4cfd0691e6dc389e
I20260928 12:30:16.420008 29167 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a390519aae4143a8ad47c4b6579722" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37915 } }
I20260928 12:30:16.420121 29167 leader_election.cc:302] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9a390519aae4143a8ad47c4b6579722; no voters:
I20260928 12:30:16.420284 29167 leader_election.cc:288] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:16.420333 29170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:16.420506 29170 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [term 1 LEADER]: Becoming Leader. State: Replica: d9a390519aae4143a8ad47c4b6579722, State: Running, Role: LEADER
I20260928 12:30:16.420653 29167 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:16.420639 29170 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a390519aae4143a8ad47c4b6579722" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37915 } }
I20260928 12:30:16.420969 29170 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a390519aae4143a8ad47c4b6579722. Latest consensus state: current_term: 1 leader_uuid: "d9a390519aae4143a8ad47c4b6579722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a390519aae4143a8ad47c4b6579722" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37915 } } }
I20260928 12:30:16.420977 29173 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9a390519aae4143a8ad47c4b6579722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9a390519aae4143a8ad47c4b6579722" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37915 } } }
I20260928 12:30:16.421054 29170 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:16.421398 29173 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:16.421443 29181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:16.421984 29181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:16.424456 29181 catalog_manager.cc:1347] Generated new cluster ID: f6a153506fbd4210b36ff77ee17cc699
I20260928 12:30:16.424508 29181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 12:30:16 dist-test-slave-cb7x krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:16.432215 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:16.420433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34683 (local address 127.7.19.126:37915)
0928 12:30:16.420584 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:16.420594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:16.421739 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 12:30:16.421987 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:16.421995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:16.422057 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:16.422201 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:16.423028 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:16.423998 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:16.425088 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:16.425736 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:16.425850 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:16.428653 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:16.428657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:16.428657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:16.428686 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:16.430591 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:16.430994 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:16.430996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:16.430997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:16.431104 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:16.431317 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:16.431318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:16.431318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:16.431603 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:16.431745 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:16.431862 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 12:30:16.431989 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 12:30:16.433300 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:16.438436 29181 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:16.439141 29181 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:16.442345 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:16.434644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56923 (local address 127.7.19.126:37915)
0928 12:30:16.434673 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:16.434675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:16.434902 (+ 227us) server_negotiation.cc:407] Connection header received
0928 12:30:16.434978 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:16.434979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:16.434988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:16.435012 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:16.435596 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:16.436388 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:16.437412 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:16.437763 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:16.437812 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:16.439273 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:16.439274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:16.439275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:16.439280 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:16.440785 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:16.441159 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:16.441160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:16.441160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:16.441208 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:16.441381 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:16.441381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:16.441381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:16.441479 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:16.441970 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:16.442174 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 12:30:16.442260 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 12:30:16.445194 29181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a390519aae4143a8ad47c4b6579722: Generated new TSK 0
I20260928 12:30:16.445371 29181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:16.449486 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29099
2026-09-28T12:30:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:16 dist-test-slave-cb7x krb5kdc[29198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:16 dist-test-slave-cb7x krb5kdc[29198](info): set up 2 sockets
Sep 28 12:30:16 dist-test-slave-cb7x krb5kdc[29198](info): commencing operation
krb5kdc: starting...
W20260928 12:30:17.480940 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:17 dist-test-slave-cb7x krb5kdc[29198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:17.599263 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:40491
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:40491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:17.678539 29211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:17.678769 29211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:17.678808 29211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:17.679949 29211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:17.680016 29211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:17.680043 29211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:17.680081 29211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:17.680106 29211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:17.681584 29211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:40491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:17.681856 29211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:17.682094 29211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:17.687368 29217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:17.687352 29219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:17.687320 29216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:17.687824 29211 server_base.cc:1034] running on GCE node
I20260928 12:30:17.688062 29211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:17.688333 29211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:17.689502 29211 hybrid_clock.cc:648] HybridClock initialized: now 1790598617689486 us; error 34 us; skew 500 ppm
Sep 28 12:30:17 dist-test-slave-cb7x krb5kdc[29198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598617, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:17.692167 29211 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:17.692596 29211 webserver.cc:460] Webserver started at http://127.7.19.126:45929/ using document root <none> and password file <none>
I20260928 12:30:17.692912 29211 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:17.692986 29211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:17.693137 29211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:17.694181 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "42863e7202fe496f9dded4bc7b5ae3b2"
format_stamp: "Formatted at 2026-09-28 12:30:17 on dist-test-slave-cb7x"
server_key: "df2d299c0dbd41b5ebb96855600d2cec"
server_key_iv: "ce5afbb5d3ff402f870ee771c27baf1d"
server_key_version: "encryptionkey@0"
I20260928 12:30:17.694538 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "42863e7202fe496f9dded4bc7b5ae3b2"
format_stamp: "Formatted at 2026-09-28 12:30:17 on dist-test-slave-cb7x"
server_key: "df2d299c0dbd41b5ebb96855600d2cec"
server_key_iv: "ce5afbb5d3ff402f870ee771c27baf1d"
server_key_version: "encryptionkey@0"
I20260928 12:30:17.695842 29211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:17.696547 29226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:17.696745 29211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:30:17.696843 29211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "42863e7202fe496f9dded4bc7b5ae3b2"
format_stamp: "Formatted at 2026-09-28 12:30:17 on dist-test-slave-cb7x"
server_key: "df2d299c0dbd41b5ebb96855600d2cec"
server_key_iv: "ce5afbb5d3ff402f870ee771c27baf1d"
server_key_version: "encryptionkey@0"
I20260928 12:30:17.696926 29211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:17.708424 29211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:17.710249 29211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:17.710410 29211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:17.714385 29211 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:40491
I20260928 12:30:17.714442 29278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:40491 every 8 connection(s)
I20260928 12:30:17.714813 29211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:17.715714 29279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:17.717963 29279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2: Bootstrap starting.
I20260928 12:30:17.718477 29279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:17.718700 29279 log.cc:824] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:17.719338 29279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2: No bootstrap required, opened a new log
I20260928 12:30:17.720894 29279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40491 } }
I20260928 12:30:17.721024 29279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:17.721045 29279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42863e7202fe496f9dded4bc7b5ae3b2, State: Initialized, Role: FOLLOWER
I20260928 12:30:17.721146 29279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40491 } }
I20260928 12:30:17.721220 29279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:17.721256 29279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:17.721338 29279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:17.722137 29279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40491 } }
I20260928 12:30:17.722254 29279 leader_election.cc:302] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42863e7202fe496f9dded4bc7b5ae3b2; no voters:
I20260928 12:30:17.722432 29279 leader_election.cc:288] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:17.722483 29282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:17.722673 29279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:17.722812 29282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 42863e7202fe496f9dded4bc7b5ae3b2, State: Running, Role: LEADER
I20260928 12:30:17.722900 29282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40491 } }
I20260928 12:30:17.723299 29284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42863e7202fe496f9dded4bc7b5ae3b2. Latest consensus state: current_term: 1 leader_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40491 } } }
I20260928 12:30:17.723308 29283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42863e7202fe496f9dded4bc7b5ae3b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 40491 } } }
I20260928 12:30:17.723388 29283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:17.723387 29284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:17.723476 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29211
I20260928 12:30:17.723575 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:17.723608 29291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:17.723837 7245 external_mini_cluster.cc:1428] Setting key f50703b627976b9fc193427f4a2706c6
I20260928 12:30:17.724895 29291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:17.726358 29291 catalog_manager.cc:1347] Generated new cluster ID: ad58ab67662d4487a5ce28b1e4be7fbe
I20260928 12:30:17.726408 29291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:17.734643 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:17.724864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57083 (local address 127.7.19.126:40491)
0928 12:30:17.728830 (+ 3966us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:17.728839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:17.728875 (+ 36us) server_negotiation.cc:407] Connection header received
0928 12:30:17.728988 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:17.728995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:17.729057 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:17.729235 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:17.729497 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:17.731383 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:17.732514 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:17.733131 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:17.733234 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:17.733960 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:17.733964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:17.733964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:17.734162 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:17.734198 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:17.734219 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:17.734278 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:17.734364 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 12:30:17.734423 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3912,"thread_start_us":70,"threads_started":1}
I20260928 12:30:17.735505 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:17.738651 29291 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:17.739110 29291 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:17.743840 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:17.736935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48305 (local address 127.7.19.126:40491)
0928 12:30:17.736969 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:17.736971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:17.738370 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 12:30:17.738418 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:17.738419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:17.738429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:17.738491 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:17.740806 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:17.741531 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:17.742503 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:17.742876 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:17.742947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:17.743592 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:17.743593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:17.743593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:17.743599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:17.743613 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:17.743617 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:17.743675 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:17.743727 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 12:30:17.743771 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 12:30:17.751508 29291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42863e7202fe496f9dded4bc7b5ae3b2: Generated new TSK 0
I20260928 12:30:17.751667 29291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:17.753520 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29211
2026-09-28T12:30:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:17 dist-test-slave-cb7x krb5kdc[29310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:17 dist-test-slave-cb7x krb5kdc[29310](info): set up 2 sockets
Sep 28 12:30:17 dist-test-slave-cb7x krb5kdc[29310](info): commencing operation
krb5kdc: starting...
W20260928 12:30:18.804872 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:18 dist-test-slave-cb7x krb5kdc[29310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:18.923148 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39919
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:19.003973 29323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:19.004177 29323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:19.004200 29323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:19.005259 29323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:19.005295 29323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:19.005309 29323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:19.005321 29323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:19.005333 29323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:19.006681 29323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:39919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:19.006857 29323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:19.007016 29323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:19.012048 29331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:19.012059 29329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:19.012084 29323 server_base.cc:1034] running on GCE node
W20260928 12:30:19.012053 29328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:19.012615 29323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:19.012964 29323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:19.014117 29323 hybrid_clock.cc:648] HybridClock initialized: now 1790598619014089 us; error 43 us; skew 500 ppm
Sep 28 12:30:19 dist-test-slave-cb7x krb5kdc[29310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598619, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:19.023001 29323 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:19.023494 29323 webserver.cc:460] Webserver started at http://127.7.19.126:43869/ using document root <none> and password file <none>
I20260928 12:30:19.023736 29323 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:19.023808 29323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:19.023962 29323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:19.025117 29323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "8884591257094619938ea2aff8284339"
format_stamp: "Formatted at 2026-09-28 12:30:19 on dist-test-slave-cb7x"
server_key: "aa7145a5b46f005ebcd1d33fd0ba5c0e"
server_key_iv: "155212fd429a1811028bc363c7791899"
server_key_version: "encryptionkey@0"
I20260928 12:30:19.025530 29323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "8884591257094619938ea2aff8284339"
format_stamp: "Formatted at 2026-09-28 12:30:19 on dist-test-slave-cb7x"
server_key: "aa7145a5b46f005ebcd1d33fd0ba5c0e"
server_key_iv: "155212fd429a1811028bc363c7791899"
server_key_version: "encryptionkey@0"
I20260928 12:30:19.026890 29323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:19.027808 29338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:19.028019 29323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:19.028136 29323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "8884591257094619938ea2aff8284339"
format_stamp: "Formatted at 2026-09-28 12:30:19 on dist-test-slave-cb7x"
server_key: "aa7145a5b46f005ebcd1d33fd0ba5c0e"
server_key_iv: "155212fd429a1811028bc363c7791899"
server_key_version: "encryptionkey@0"
I20260928 12:30:19.028232 29323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:19.031605 29323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:19.033869 29323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:19.034019 29323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:19.038177 29323 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:39919
I20260928 12:30:19.038229 29390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:39919 every 8 connection(s)
I20260928 12:30:19.038520 29323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:19.039448 29391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:19.041723 29391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339: Bootstrap starting.
I20260928 12:30:19.042279 29391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:19.042550 29391 log.cc:824] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:19.043216 29391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339: No bootstrap required, opened a new log
I20260928 12:30:19.044793 29391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8884591257094619938ea2aff8284339" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39919 } }
I20260928 12:30:19.044920 29391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 12:30:19.044942 29391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8884591257094619938ea2aff8284339, State: Initialized, Role: FOLLOWER
I20260928 12:30:19.045091 29391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8884591257094619938ea2aff8284339" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39919 } }
I20260928 12:30:19.045149 29391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 12:30:19.045207 29391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 12:30:19.045261 29391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:19.046032 29391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8884591257094619938ea2aff8284339" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39919 } }
I20260928 12:30:19.046139 29391 leader_election.cc:302] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8884591257094619938ea2aff8284339; no voters:
I20260928 12:30:19.046300 29391 leader_election.cc:288] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 12:30:19.046339 29394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:19.046515 29394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [term 1 LEADER]: Becoming Leader. State: Replica: 8884591257094619938ea2aff8284339, State: Running, Role: LEADER
I20260928 12:30:19.046548 29391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:19.046694 29394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8884591257094619938ea2aff8284339" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39919 } }
I20260928 12:30:19.047081 29396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8884591257094619938ea2aff8284339. Latest consensus state: current_term: 1 leader_uuid: "8884591257094619938ea2aff8284339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8884591257094619938ea2aff8284339" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39919 } } }
I20260928 12:30:19.047187 29396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:19.047380 29403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:19.047070 29395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8884591257094619938ea2aff8284339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8884591257094619938ea2aff8284339" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 39919 } } }
I20260928 12:30:19.047852 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29323
I20260928 12:30:19.047966 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:19.048028 29403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:19.048170 7245 external_mini_cluster.cc:1428] Setting key 805b6f8f9e452a7496fbf915fa907624
I20260928 12:30:19.047631 29395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:19.049932 29403 catalog_manager.cc:1347] Generated new cluster ID: e0c5872f61374666baafc783eb66e9af
I20260928 12:30:19.049985 29403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:19.050655 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:19.048847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54255 (local address 127.7.19.126:39919)
0928 12:30:19.049035 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:19.049043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:19.049074 (+ 31us) server_negotiation.cc:407] Connection header received
0928 12:30:19.049181 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:19.049187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:19.049215 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:19.049355 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:19.050100 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:19.050104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:19.050104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:19.050108 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:19.050147 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:19.050163 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:19.050216 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:19.050434 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 12:30:19.050473 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260928 12:30:19.051409 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:19.053961 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:19.052916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58245 (local address 127.7.19.126:39919)
0928 12:30:19.052945 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:19.052947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:19.053102 (+ 155us) server_negotiation.cc:407] Connection header received
0928 12:30:19.053131 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:19.053132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:19.053137 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:19.053205 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:19.053754 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:19.053755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:19.053755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 12:30:19.053759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:19.053773 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 12:30:19.053777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:19.053837 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:19.053888 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 12:30:19.053902 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 12:30:19.069797 29403 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:19.069979 29403 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:19.076550 29403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8884591257094619938ea2aff8284339: Generated new TSK 0
I20260928 12:30:19.076771 29403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:19.081812 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29323
2026-09-28T12:30:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1328 ms)
[----------] 18 tests from AuthTokenIssuingTest (23863 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:19 dist-test-slave-cb7x krb5kdc[29422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:19 dist-test-slave-cb7x krb5kdc[29422](info): set up 2 sockets
Sep 28 12:30:19 dist-test-slave-cb7x krb5kdc[29422](info): commencing operation
krb5kdc: starting...
W20260928 12:30:20.129288 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:20.247598 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37527
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37527,127.7.19.125:39951,127.7.19.124:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:20.335381 29435 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:30:20.335602 29435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:20.335621 29435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:20.335680 29435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:20.337003 29435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:20.337049 29435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:20.337064 29435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:20.337121 29435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:20.337142 29435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:20.338864 29435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37527,127.7.19.125:39951,127.7.19.124:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:20.339143 29435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:20.339387 29435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:20.344408 29443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:20.344483 29440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:20.344594 29435 server_base.cc:1034] running on GCE node
W20260928 12:30:20.344473 29441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:20.344898 29435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:20.345180 29435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:20.346354 29435 hybrid_clock.cc:648] HybridClock initialized: now 1790598620346338 us; error 46 us; skew 500 ppm
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:20.348773 29435 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:20.349248 29435 webserver.cc:460] Webserver started at http://127.7.19.126:40059/ using document root <none> and password file <none>
I20260928 12:30:20.349486 29435 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:20.349555 29435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:20.349689 29435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:20.350705 29435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "4f1b32aab4e71efa83f0362d9e769eec"
server_key_iv: "7c5a9340faea5d3288edb54763c496eb"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.351083 29435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "4f1b32aab4e71efa83f0362d9e769eec"
server_key_iv: "7c5a9340faea5d3288edb54763c496eb"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.352350 29435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 12:30:20.353113 29450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:20.353289 29435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:20.353379 29435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "4f1b32aab4e71efa83f0362d9e769eec"
server_key_iv: "7c5a9340faea5d3288edb54763c496eb"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.353461 29435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:20.387404 29435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:20.389480 29435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:20.389662 29435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:20.393759 29435 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:37527
I20260928 12:30:20.393796 29502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:37527 every 8 connection(s)
I20260928 12:30:20.394171 29435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:20.395134 29503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:20.396759 29503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:20.402484 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29435
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:20.402626 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:20.402858 7245 external_mini_cluster.cc:1428] Setting key 653118809ecd34d0a9da1c07b45cb4c6
I20260928 12:30:20.407003 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.397397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37527 (local address 127.0.0.1:33964)
0928 12:30:20.397576 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.397619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.398063 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.398372 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.398375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.398616 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.398772 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.398776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.399780 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.399781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.400549 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.400551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.401063 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.401064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.401123 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.401599 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.401607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.403241 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.405551 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.405557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.405558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.405749 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.406011 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.406011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.406012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.406090 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.406553 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.406558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.406615 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.406691 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.406839 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260928 12:30:20.407068 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.397474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.7.19.126:37527)
0928 12:30:20.397643 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.397649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.397698 (+ 49us) server_negotiation.cc:407] Connection header received
0928 12:30:20.398236 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.398238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.398255 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.398342 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.398870 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.399699 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.400633 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.400961 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.401047 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.403356 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.403363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.403363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.403370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.405471 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.405832 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.405835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.405835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.405942 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.406158 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.406159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.406160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.406472 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.406568 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.406757 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.406922 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":130,"thread_start_us":166,"threads_started":1}
I20260928 12:30:20.407796 29503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:20.408159 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.408024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53104)
0928 12:30:20.408076 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.408128 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:30:20.408322 29451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
I20260928 12:30:20.408803 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.408693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53110)
0928 12:30:20.408745 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.408776 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 12:30:20.408921 29503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:39951: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.7.19.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab.
Entry for principal kudu/127.7.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab.
I20260928 12:30:20.427940 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:39951
--webserver_interface=127.7.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab
--principal=kudu/127.7.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37527,127.7.19.125:39951,127.7.19.124:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:30:20.458143 29503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } attempt: 1
I20260928 12:30:20.458619 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.458467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53122)
0928 12:30:20.458532 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.458582 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 12:30:20.458976 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.458890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53138)
0928 12:30:20.458931 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.458958 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:30:20.459092 29503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:39951: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
W20260928 12:30:20.507238 29510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:30:20.507443 29510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:20.507460 29510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:20.507480 29510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:20.508630 29510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:20.508692 29510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:20.508708 29510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:20.508762 29510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:20.508782 29510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:20.510224 29510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37527,127.7.19.125:39951,127.7.19.124:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab
--principal=kudu/127.7.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:39951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:20.510475 29510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:20.510696 29510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 12:30:20.515592 29510 server_base.cc:1034] running on GCE node
W20260928 12:30:20.515549 29515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:20.515587 29516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:20.515637 29518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:20.516039 29510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:20.516283 29510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:20.517480 29510 hybrid_clock.cc:648] HybridClock initialized: now 1790598620517472 us; error 36 us; skew 500 ppm
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:20.519989 29510 init.cc:377] Logged in from keytab as kudu/127.7.19.125@KRBTEST.COM (short username kudu)
I20260928 12:30:20.520489 29510 webserver.cc:460] Webserver started at http://127.7.19.125:36327/ using document root <none> and password file <none>
I20260928 12:30:20.520789 29510 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:20.520860 29510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:20.521023 29510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:20.522126 29510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data/instance:
uuid: "52c8130ab8e44b5c82d658038793e34d"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "a11d08a8f2d8abd2658c8ac65b89da25"
server_key_iv: "d555a72d7683eac660a63bafbfaaf2b4"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.522521 29510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal/instance:
uuid: "52c8130ab8e44b5c82d658038793e34d"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "a11d08a8f2d8abd2658c8ac65b89da25"
server_key_iv: "d555a72d7683eac660a63bafbfaaf2b4"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.523833 29510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 12:30:20.524559 29525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:20.524760 29510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 12:30:20.524864 29510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal
uuid: "52c8130ab8e44b5c82d658038793e34d"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "a11d08a8f2d8abd2658c8ac65b89da25"
server_key_iv: "d555a72d7683eac660a63bafbfaaf2b4"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.524946 29510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:20.527269 29503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } attempt: 2
I20260928 12:30:20.527679 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.527521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53140)
0928 12:30:20.527603 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.527642 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 12:30:20.528030 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.527928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53146)
0928 12:30:20.527976 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.528007 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 12:30:20.528138 29503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:39951: Network error: Client connection negotiation failed: client connection to 127.7.19.125:39951: connect: Connection refused (error 111)
I20260928 12:30:20.539315 29510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:20.541133 29510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:20.541239 29510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:20.545023 29510 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.125:39951
I20260928 12:30:20.545071 29577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.125:39951 every 8 connection(s)
I20260928 12:30:20.545432 29510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
I20260928 12:30:20.546360 29578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:20.548267 29578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:20.552419 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29510
I20260928 12:30:20.552501 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-1/wal/instance
I20260928 12:30:20.552975 7245 external_mini_cluster.cc:1428] Setting key 8b372282d8f281f84fa6a0ec71a3f00f
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:20.557817 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.549073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.7.19.126:37527)
0928 12:30:20.549114 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.549117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.549351 (+ 234us) server_negotiation.cc:407] Connection header received
0928 12:30:20.549814 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.549816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.549825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.549880 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.550457 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.550937 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.551965 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.552160 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.552210 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.554654 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.554655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.554655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.554662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.556144 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.556718 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.556719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.556719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.556784 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.557113 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.557114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.557115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.557232 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.557287 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.557684 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.557750 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:30:20.558096 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37527 (local address 127.0.0.1:33966)
0928 12:30:20.549216 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.549258 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.549730 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.549935 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.549938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.550194 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.550386 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.550390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.551020 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.551021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.551887 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.551890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.552250 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.552251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.552356 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.552896 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.552904 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.554548 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.556244 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.556250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.556252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.556606 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.556856 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.556857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.556858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.557020 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.557307 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.557315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.557502 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.557663 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.557853 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 12:30:20.558703 29578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.19.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.124@KRBTEST.COM" created.
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:20.569219 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.558955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53154)
0928 12:30:20.559013 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.559026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.559074 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.559273 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.559274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.559449 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.559524 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.559525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.560356 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.560357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.561325 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.561327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.561787 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.561788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.561886 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.562287 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.562293 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.565864 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.568039 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.568040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.568040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.568207 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.568481 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.568481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.568482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.568544 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.568866 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.568867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.568904 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.568996 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.569106 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:30:20.569446 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.558989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.7.19.125:39951)
0928 12:30:20.559125 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.559130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.559147 (+ 17us) server_negotiation.cc:407] Connection header received
0928 12:30:20.559175 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.559176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.559186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.559246 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.559606 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.560265 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.561412 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.561699 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.561755 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.565949 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.565956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.565957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.565963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.567949 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.568337 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.568340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.568341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.568401 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.568611 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.568612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.568612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.568792 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.568890 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.569029 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.569223 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260928 12:30:20.570111 29578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:20.570475 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.570356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33034)
0928 12:30:20.570403 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.570448 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 12:30:20.570603 29529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
I20260928 12:30:20.571094 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.571002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33042)
0928 12:30:20.571049 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.571073 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:30:20.571202 29578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44867: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab.
Entry for principal kudu/127.7.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab.
I20260928 12:30:20.583904 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:44867
--webserver_interface=127.7.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab
--principal=kudu/127.7.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:37527,127.7.19.125:39951,127.7.19.124:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:30:20.619297 29503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } attempt: 3
I20260928 12:30:20.620421 29578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } attempt: 1
I20260928 12:30:20.620885 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.620747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33046)
0928 12:30:20.620807 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.620847 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:30:20.621279 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.621191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33060)
0928 12:30:20.621229 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.621252 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 12:30:20.621354 29578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44867: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:20.626809 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.619766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.7.19.125:39951)
0928 12:30:20.619836 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.619838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.619865 (+ 27us) server_negotiation.cc:407] Connection header received
0928 12:30:20.619954 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.619955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.619973 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.620040 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.620387 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.621135 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.621962 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.622216 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.622315 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.624225 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.624226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.624227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.624234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.625643 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.626000 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.626001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.626002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.626061 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.626270 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.626271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.626271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.626395 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.626480 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.626634 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.626710 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 12:30:20.626816 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.619696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53170)
0928 12:30:20.619766 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.619776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.619885 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.620047 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.620048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.620216 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.620282 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.620284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.621235 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.621236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.621869 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.621871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.622295 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.622296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.622371 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.622901 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.622907 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.624134 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.625733 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.625734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.625734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.625913 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.626133 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.626134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.626134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.626196 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.626471 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.626472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.626519 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.626609 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.626710 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 12:30:20.627168 29503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:20.627552 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.627465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33062)
0928 12:30:20.627498 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.627525 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 12:30:20.627950 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.627866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33076)
0928 12:30:20.627912 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.627935 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 12:30:20.628054 29503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44867: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
I20260928 12:30:20.659190 29503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } attempt: 1
I20260928 12:30:20.659678 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.659543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33090)
0928 12:30:20.659610 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.659650 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 12:30:20.660063 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.659981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33094)
0928 12:30:20.660023 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.660047 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:30:20.660169 29503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44867: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
W20260928 12:30:20.666193 29585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:30:20.666384 29585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:20.666401 29585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:20.666446 29585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:20.667564 29585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:20.667603 29585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:20.667616 29585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:20.667655 29585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:20.667681 29585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:20.669229 29585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:37527,127.7.19.125:39951,127.7.19.124:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab
--principal=kudu/127.7.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:20.669480 29585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:20.669708 29585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:20.674674 29590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:20.674649 29593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:20.674687 29585 server_base.cc:1034] running on GCE node
W20260928 12:30:20.674646 29591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:20.675143 29585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:20.675452 29585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:20.676621 29585 hybrid_clock.cc:648] HybridClock initialized: now 1790598620676596 us; error 35 us; skew 500 ppm
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:20.679220 29585 init.cc:377] Logged in from keytab as kudu/127.7.19.124@KRBTEST.COM (short username kudu)
I20260928 12:30:20.679662 29585 webserver.cc:460] Webserver started at http://127.7.19.124:32999/ using document root <none> and password file <none>
I20260928 12:30:20.679886 29585 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:20.679953 29585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:20.680104 29585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:20.681231 29585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data/instance:
uuid: "217a981379da447aa0c8ec5eeed0fd59"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "b6a9a6dfbd5c728e9ea6ca2e1b4a6524"
server_key_iv: "f17e0856c3a0699f7cc98ab30d29df23"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.681608 29585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal/instance:
uuid: "217a981379da447aa0c8ec5eeed0fd59"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "b6a9a6dfbd5c728e9ea6ca2e1b4a6524"
server_key_iv: "f17e0856c3a0699f7cc98ab30d29df23"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.682927 29585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:20.683636 29600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:20.683825 29585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:20.683926 29585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal
uuid: "217a981379da447aa0c8ec5eeed0fd59"
format_stamp: "Formatted at 2026-09-28 12:30:20 on dist-test-slave-cb7x"
server_key: "b6a9a6dfbd5c728e9ea6ca2e1b4a6524"
server_key_iv: "f17e0856c3a0699f7cc98ab30d29df23"
server_key_version: "encryptionkey@0"
I20260928 12:30:20.684005 29585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:20.689483 29578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } attempt: 2
I20260928 12:30:20.689988 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.689866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33098)
0928 12:30:20.689921 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.689958 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:30:20.690371 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.690290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33110)
0928 12:30:20.690332 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.690355 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 12:30:20.690459 29578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44867: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44867: connect: Connection refused (error 111)
I20260928 12:30:20.701115 29585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:20.703456 29585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:20.703572 29585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:20.707396 29585 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.124:44867
I20260928 12:30:20.707448 29652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.124:44867 every 8 connection(s)
I20260928 12:30:20.707799 29585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
I20260928 12:30:20.708467 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29585
I20260928 12:30:20.708546 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790598357699237-7245-0/minicluster-data/master-2/wal/instance
I20260928 12:30:20.708792 7245 external_mini_cluster.cc:1428] Setting key 9c838cf5977658a4b48ce00431604f0e
I20260928 12:30:20.708894 29653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:20.710901 29653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:20.720894 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.709315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.7.19.124:44867)
0928 12:30:20.709510 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.709519 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.709557 (+ 38us) server_negotiation.cc:407] Connection header received
0928 12:30:20.709960 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.709966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.710019 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.710167 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.710770 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.711875 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.712651 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.713272 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.713395 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.716241 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.716250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.716250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.716258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.718483 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.718876 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.718879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.718880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.719045 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.719324 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.719325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.719326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.719790 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.720366 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.720438 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.720563 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 12:30:20.721356 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.711546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.7.19.126:37527)
0928 12:30:20.711612 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.711614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.711754 (+ 140us) server_negotiation.cc:407] Connection header received
0928 12:30:20.711852 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.711854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.711866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.711932 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.713209 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.713841 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.715015 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.715198 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.715296 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.718157 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.718158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.718159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.718166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.719785 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.720374 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.720375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.720376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.720475 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.720730 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.720731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.720731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.720903 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.720951 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.721181 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.721260 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 12:30:20.721801 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.711490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:37527 (local address 127.0.0.1:33970)
0928 12:30:20.711649 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.711688 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.711761 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.712713 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.712715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.713039 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.713111 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.713112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.713945 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.713946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.714923 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.714926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.715329 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.715377 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.715461 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.716014 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.716026 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.717990 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.719891 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.719898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.719900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.720172 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.720536 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.720537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.720537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.720633 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.721000 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.721006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.721052 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.721465 (+ 413us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.721664 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260928 12:30:20.723011 29653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:20.729628 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.721897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.7.19.126:37527)
0928 12:30:20.721934 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.721936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.721943 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:30:20.721961 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.721961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.721971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.722022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.722619 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.723176 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.724007 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.724216 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.724301 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.726563 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.726564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.726564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.726570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.728334 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.728662 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.728663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.728664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.728744 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.729008 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.729008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.729009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.729134 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.729316 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.729411 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.729485 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:30:20.729914 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.723465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:39951 (local address 127.0.0.1:53184)
0928 12:30:20.723520 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.723530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.723571 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.723698 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.723698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.723837 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.723879 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.723881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.724494 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.724495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.725047 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.725049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.725437 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.725438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.725504 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.725886 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.725890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.727116 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.728859 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.728860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.728861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.729081 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.729273 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.729274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.729274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.729343 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.729613 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.729613 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.729655 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.729741 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.729827 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:30:20.729967 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.723560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.7.19.125:39951)
0928 12:30:20.723605 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.723607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.723615 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:20.723639 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.723640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.723648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.723696 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.723958 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.724412 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.725128 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.725351 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.725437 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.727201 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.727203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.727203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.727211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.728752 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.729148 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.729149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.729149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.729205 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.729410 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.729411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.729411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.729541 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.729608 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.729772 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.729871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:30:20.730475 29653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:20.735311 29503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } attempt: 2
I20260928 12:30:20.737236 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.730714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33116)
0928 12:30:20.730811 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.730826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.730862 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.730977 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.730978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.731082 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.731107 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.731108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.731828 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.731829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.732312 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.732313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.732757 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.732758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.732824 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.733190 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.733195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.734389 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.736237 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.736237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.736238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.736460 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.736657 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.736657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.736657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.736731 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.736986 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.736987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.737017 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.737070 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.737146 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 12:30:20.737528 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.730767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.7.19.124:44867)
0928 12:30:20.730804 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.730806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.730854 (+ 48us) server_negotiation.cc:407] Connection header received
0928 12:30:20.730899 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.730900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.730910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.730976 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.731177 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.731674 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.732386 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.732633 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.732752 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.734468 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.734470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.734470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.734478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.736136 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.736544 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.736545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.736546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.736598 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.736806 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.736807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.736807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.736912 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.736967 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.737389 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.737459 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:20.738545 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.730429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.7.19.125:39951)
0928 12:30:20.730463 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.730465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.730472 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:30:20.730523 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.730524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.730530 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.730599 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.731207 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.731624 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.732434 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.733169 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.733240 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.735561 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.735563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.735563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.735569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.737157 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.737545 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.737546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.737546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.737719 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.738035 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.738036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.738036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.738160 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.738315 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.738397 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.738468 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:30:20.739326 29653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: Bootstrap starting.
I20260928 12:30:20.739868 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:20.740077 29653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: Neither blocks nor log segments found. Creating new log.
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:30:20.740427 29653 log.cc:824] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:20.741065 29477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33974
I20260928 12:30:20.742635 29653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: No bootstrap required, opened a new log
I20260928 12:30:20.743631 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.735554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33132)
0928 12:30:20.735603 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.735613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.735646 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.735892 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.735893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.736057 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.736101 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.736103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.737453 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.737454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.738222 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.738225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.738933 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.738933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.738992 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.739387 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.739392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.740816 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.742557 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.742558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.742558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.742782 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.742977 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.742978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.742979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.743039 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.743338 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.743338 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.743379 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.743447 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.743536 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:30:20.743695 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.735603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.7.19.124:44867)
0928 12:30:20.735761 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.735764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.735773 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:20.735793 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.735794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.735811 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.735886 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.736185 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.737358 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.738321 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.738734 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.738826 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.740931 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.740932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.740932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.740939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.742474 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.742846 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.742847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.742847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.742905 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.743106 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.743107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.743107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.743257 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.743327 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.743498 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.743574 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260928 12:30:20.744771 29653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.744939 29653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217a981379da447aa0c8ec5eeed0fd59, State: Initialized, Role: FOLLOWER
I20260928 12:30:20.745031 29653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.745151 29503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2: Bootstrap starting.
I20260928 12:30:20.745378 29653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:20.745906 29503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:20.746251 29503 log.cc:824] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:20.746861 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.747068 29503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2: No bootstrap required, opened a new log
I20260928 12:30:20.747174 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:30:20.747607 29672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:20.747663 29672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:20.748375 29503 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.748507 29503 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcac0f1e39a4dfcaf7b12217f9905b2, State: Initialized, Role: FOLLOWER
I20260928 12:30:20.748579 29503 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.748890 29503 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:20.748871 29674 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.748958 29674 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:20.749019 29477 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 12:30:20.749068 29477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:20.749619 29477 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.749873 29477 leader_election.cc:288] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [CANDIDATE]: Term 1 election: Requested vote from peers 52c8130ab8e44b5c82d658038793e34d (127.7.19.125:39951), 217a981379da447aa0c8ec5eeed0fd59 (127.7.19.124:44867)
I20260928 12:30:20.750327 29627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "217a981379da447aa0c8ec5eeed0fd59"
I20260928 12:30:20.750434 29627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [term 0 FOLLOWER]: Advancing to term 1
W20260928 12:30:20.750768 29685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:20.750869 29685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:20.751359 29627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdcac0f1e39a4dfcaf7b12217f9905b2 in term 1.
I20260928 12:30:20.751878 29454 leader_election.cc:302] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 217a981379da447aa0c8ec5eeed0fd59, fdcac0f1e39a4dfcaf7b12217f9905b2; no voters:
I20260928 12:30:20.752113 29674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:20.752326 29674 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [term 1 LEADER]: Becoming Leader. State: Replica: fdcac0f1e39a4dfcaf7b12217f9905b2, State: Running, Role: LEADER
I20260928 12:30:20.752493 29674 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.753152 29674 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcac0f1e39a4dfcaf7b12217f9905b2. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.753252 29674 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:20.753515 29693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:20.753820 29693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:20.751775 29585 master_runner.cc:186] Error getting master registration for 127.7.19.125:39951: OK, instance_id { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" instance_seqno: 1790598620543930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:30:20.753952 29585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:30:20.752646 29435 master_runner.cc:186] Error getting master registration for 127.7.19.125:39951: OK, instance_id { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" instance_seqno: 1790598620543930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:30:20.755273 29435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:30:20.756099 29627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [term 1 FOLLOWER]: Refusing update from remote peer fdcac0f1e39a4dfcaf7b12217f9905b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:20.756443 29691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:20.757783 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcac0f1e39a4dfcaf7b12217f9905b2. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.758060 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:20.758940 29691 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.759048 29691 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:20.759140 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.759214 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:20.760344 29694 mvcc.cc:205] Tried to move back new op lower bound from 7334291950614749184 to 7334291950603509760. Current Snapshot: MvccSnapshot[applied={T|T < 7334291950614749184}]
I20260928 12:30:20.760819 29697 mvcc.cc:205] Tried to move back new op lower bound from 7334291950614749184 to 7334291950603509760. Current Snapshot: MvccSnapshot[applied={T|T < 7334291950614749184}]
I20260928 12:30:20.763242 29693 catalog_manager.cc:1347] Generated new cluster ID: c2b19b5120884ecdada9de083ba149dc
I20260928 12:30:20.763293 29693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:20.766223 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.756999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.7.19.124:44867)
0928 12:30:20.757042 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.757044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.757335 (+ 291us) server_negotiation.cc:407] Connection header received
0928 12:30:20.757436 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.757437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.757449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.757482 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.758273 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.759041 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.759868 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.760204 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.760277 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.762632 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.762634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.762634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.762642 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.764461 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.765104 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.765105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.765106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.765178 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.765515 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.765516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.765516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.765642 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.765729 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.766054 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.766142 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:30:20.767287 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.757440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.7.19.126:37527)
0928 12:30:20.757476 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.757478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.758377 (+ 899us) server_negotiation.cc:407] Connection header received
0928 12:30:20.758477 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.758478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.758488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.758516 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.760292 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.760976 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.762075 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.762418 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.762459 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.764273 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.764274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.764275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.764281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.765915 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.766422 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.766423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.766423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.766470 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.766708 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.766709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.766709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.766818 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.766874 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.767144 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.767209 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:20.776191 29693 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:20.776925 29693 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:20.780728 29693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2: Generated new TSK 0
I20260928 12:30:20.780901 29693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 12:30:20.781157 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.757627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.7.19.125:39951)
0928 12:30:20.757663 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.757665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.764784 (+ 7119us) server_negotiation.cc:407] Connection header received
0928 12:30:20.764887 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.764889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.764901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.764932 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.765577 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.766259 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.767390 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.767682 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.767740 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.778153 (+ 10413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.778155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.778155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.778162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.779757 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.780208 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.780209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.780210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.780294 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.780522 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.780523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.780523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.780651 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.780764 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.780985 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.781064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:20.781585 29578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } attempt: 3
Sep 28 12:30:20 dist-test-slave-cb7x krb5kdc[29422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598620, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:30:20.789435 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.781800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44867 (local address 127.0.0.1:33146)
0928 12:30:20.781856 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:20.781867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:20.781902 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:20.782040 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:20.782041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:20.782188 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:20.782229 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.782230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.783132 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.783133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.783910 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:20.783912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.784371 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.784373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:20.784440 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.784842 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:20.784847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:20.786382 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:20.788423 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.788424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.788425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.788596 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.788827 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:20.788828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:20.788828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:20.788885 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:20.789132 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:20.789133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:20.789177 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:30:20.789239 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 12:30:20.789328 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 12:30:20.789722 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:20.781867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.7.19.124:44867)
0928 12:30:20.781902 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.781904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.781913 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:20.781953 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.781954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.781962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.782020 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.782302 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.783035 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.784002 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:20.784296 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:20.784360 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:20.786483 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:20.786484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:20.786484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:20.786492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:20.788329 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.788700 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.788701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.788701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.788757 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:20.788947 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:20.788948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:20.788948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:20.789054 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:20.789127 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:20.789544 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 12:30:20.789628 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 12:30:20.791329 29578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: Bootstrap starting.
I20260928 12:30:20.792017 29578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:20.792291 29578 log.cc:824] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:20.793093 29578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: No bootstrap required, opened a new log
I20260928 12:30:20.795055 29578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.795200 29578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c8130ab8e44b5c82d658038793e34d, State: Initialized, Role: FOLLOWER
I20260928 12:30:20.795282 29578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } }
I20260928 12:30:20.795588 29578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:20.795655 29552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "52c8130ab8e44b5c82d658038793e34d"
I20260928 12:30:20.795755 29552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:20.796689 29552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdcac0f1e39a4dfcaf7b12217f9905b2 in term 1.
I20260928 12:30:20.797029 29551 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [term 1 FOLLOWER]: Refusing update from remote peer fdcac0f1e39a4dfcaf7b12217f9905b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:20.797376 29674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:20.799018 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.799103 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:20.799248 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcac0f1e39a4dfcaf7b12217f9905b2. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.799317 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:20.799357 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:20.799408 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d [sys.catalog]: This master's current role is: FOLLOWER
W20260928 12:30:20.801213 29715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:20.801272 29715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:20.808283 29717 mvcc.cc:205] Tried to move back new op lower bound from 7334291950712934400 to 7334291950603509760. Current Snapshot: MvccSnapshot[applied={T|T < 7334291950614749184}]
W20260928 12:30:20.808733 29506 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.807787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.7.19.126:37527)
0928 12:30:20.807864 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.807866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.808047 (+ 181us) server_negotiation.cc:407] Connection header received
0928 12:30:20.808149 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.808164 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.808175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.808205 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.808634 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33988: BlockingRecv error: recv got EOF from 127.0.0.1:33988 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 12:30:20.809094 29658 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.807448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.7.19.124:44867)
0928 12:30:20.807501 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.807503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.808489 (+ 986us) server_negotiation.cc:407] Connection header received
0928 12:30:20.808585 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.808587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.808596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.808626 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.809026 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33150: BlockingRecv error: recv got EOF from 127.0.0.1:33150 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:30:20.809449 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.807825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.7.19.125:39951)
0928 12:30:20.807892 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.807894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.808963 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 12:30:20.809055 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.809056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.809066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.809096 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.809402 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53196: BlockingRecv error: recv got EOF from 127.0.0.1:53196 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 12:30:20.811158 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.810702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.7.19.125:39951)
0928 12:30:20.810736 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.810738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.810747 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:20.810776 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.810776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.810790 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.810839 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.811106 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53200: BlockingRecv error: recv got EOF from 127.0.0.1:53200 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:20.813741 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.813154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.7.19.125:39951)
0928 12:30:20.813184 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.813186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.813284 (+ 98us) server_negotiation.cc:407] Connection header received
0928 12:30:20.813372 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.813373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.813382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.813409 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.813700 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53210: BlockingRecv error: recv got EOF from 127.0.0.1:53210 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:20.817689 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.816944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.7.19.125:39951)
0928 12:30:20.816985 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.816986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.817195 (+ 209us) server_negotiation.cc:407] Connection header received
0928 12:30:20.817290 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.817290 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.817300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.817329 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.817649 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53212: BlockingRecv error: recv got EOF from 127.0.0.1:53212 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:20.823552 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.822855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.7.19.125:39951)
0928 12:30:20.822888 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.822890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.823093 (+ 203us) server_negotiation.cc:407] Connection header received
0928 12:30:20.823182 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.823183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.823191 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.823218 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.823508 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53218: BlockingRecv error: recv got EOF from 127.0.0.1:53218 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:20.833468 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.832724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.7.19.125:39951)
0928 12:30:20.832756 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.832758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.832988 (+ 230us) server_negotiation.cc:407] Connection header received
0928 12:30:20.833081 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.833082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.833091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.833119 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.833413 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53228: BlockingRecv error: recv got EOF from 127.0.0.1:53228 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:20.851419 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.850683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.7.19.125:39951)
0928 12:30:20.850714 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.850716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.850934 (+ 218us) server_negotiation.cc:407] Connection header received
0928 12:30:20.851028 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.851029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.851039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.851066 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.851374 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53234: BlockingRecv error: recv got EOF from 127.0.0.1:53234 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:20.885864 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.884903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.7.19.125:39951)
0928 12:30:20.884949 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.884951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.885234 (+ 283us) server_negotiation.cc:407] Connection header received
0928 12:30:20.885347 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.885348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.885360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.885391 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.885804 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53238: BlockingRecv error: recv got EOF from 127.0.0.1:53238 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 12:30:20.952030 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:20.951242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.7.19.125:39951)
0928 12:30:20.951289 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:20.951292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:20.951437 (+ 145us) server_negotiation.cc:407] Connection header received
0928 12:30:20.951510 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:20.951511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:20.951522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:20.951597 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:20.951962 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53254: BlockingRecv error: recv got EOF from 127.0.0.1:53254 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 12:30:21.083297 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:21.082408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.7.19.125:39951)
0928 12:30:21.082485 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.082488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.082601 (+ 113us) server_negotiation.cc:407] Connection header received
0928 12:30:21.082746 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.082747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.082760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.082837 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:21.083223 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53258: BlockingRecv error: recv got EOF from 127.0.0.1:53258 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 12:30:21.235198 29674 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdcac0f1e39a4dfcaf7b12217f9905b2" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 37527 } } peers { permanent_uuid: "52c8130ab8e44b5c82d658038793e34d" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 39951 } } peers { permanent_uuid: "217a981379da447aa0c8ec5eeed0fd59" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44867 } } }
I20260928 12:30:21.235371 29674 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcac0f1e39a4dfcaf7b12217f9905b2 [sys.catalog]: This master's current role is: LEADER
W20260928 12:30:21.341187 29582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:21.340451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.7.19.125:39951)
0928 12:30:21.340519 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.340521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.340574 (+ 53us) server_negotiation.cc:407] Connection header received
0928 12:30:21.340625 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.340626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.340730 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.340786 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:21.341125 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53266: BlockingRecv error: recv got EOF from 127.0.0.1:53266 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 12:30:21.748092 29672 catalog_manager.cc:1259] Loaded cluster ID: c2b19b5120884ecdada9de083ba149dc
I20260928 12:30:21.748185 29672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: loading cluster ID for follower catalog manager: success
I20260928 12:30:21.749089 29672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: acquiring CA information for follower catalog manager: success
I20260928 12:30:21.749396 29672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 217a981379da447aa0c8ec5eeed0fd59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:30:21.801681 29715 catalog_manager.cc:1259] Loaded cluster ID: c2b19b5120884ecdada9de083ba149dc
I20260928 12:30:21.801755 29715 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: loading cluster ID for follower catalog manager: success
I20260928 12:30:21.802503 29715 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: acquiring CA information for follower catalog manager: success
I20260928 12:30:21.802773 29715 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52c8130ab8e44b5c82d658038793e34d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:30:21.857712 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:21.854342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.7.19.125:39951)
0928 12:30:21.854495 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.854498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.854630 (+ 132us) server_negotiation.cc:407] Connection header received
0928 12:30:21.854754 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.854755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.854821 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.854936 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:21.855125 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.855754 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.856799 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.857029 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.857117 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:21.857309 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:21.857430 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:21.857531 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:21.857557 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 12:30:21.857603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260928 12:30:21.862254 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:21.859739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.7.19.124:44867)
0928 12:30:21.859831 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.859832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.859840 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:21.859919 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.859919 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.859957 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.860030 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:21.860168 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.860642 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.861494 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.861681 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.861741 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:21.861908 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:21.862005 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:21.862063 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:21.862175 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 12:30:21.862193 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":43,"threads_started":1}
I20260928 12:30:21.866986 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:21.864148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.7.19.124:44867)
0928 12:30:21.864185 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.864187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.864344 (+ 157us) server_negotiation.cc:407] Connection header received
0928 12:30:21.864453 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.864454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.864465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.864514 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:21.864654 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.865126 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.866111 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.866439 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.866519 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:21.866689 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:21.866723 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:21.866777 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:21.866864 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:30:21.866894 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 12:30:21.867285 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:21.864350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.7.19.125:39951)
0928 12:30:21.864390 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.864392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.864399 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:30:21.864424 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.864425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.864434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.864493 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:21.864609 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.865463 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.866364 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.866616 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.866703 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:21.866972 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:21.867011 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:21.867042 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:21.867185 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 12:30:21.867215 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 12:30:21.867712 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:21.864104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.7.19.126:37527)
0928 12:30:21.864212 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:21.864214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:21.864285 (+ 71us) server_negotiation.cc:407] Connection header received
0928 12:30:21.864425 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:21.864426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:21.864481 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:21.864541 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:21.864781 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.865600 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.866611 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:21.866967 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:21.867033 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:21.867293 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:21.867411 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:21.867458 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:21.867586 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 12:30:21.867618 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260928 12:30:21.869275 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29435
I20260928 12:30:21.874717 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29510
I20260928 12:30:21.879256 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29585
2026-09-28T12:30:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2799 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 12:30:21 dist-test-slave-cb7x krb5kdc[29806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 12:30:21 dist-test-slave-cb7x krb5kdc[29806](info): set up 2 sockets
Sep 28 12:30:21 dist-test-slave-cb7x krb5kdc[29806](info): commencing operation
krb5kdc: starting...
W20260928 12:30:22.925099 7245 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/test-user.keytab.
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T12:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T12:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.19.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
Entry for principal kudu/127.7.19.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab.
I20260928 12:30:23.042698 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46739
--webserver_interface=127.7.19.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46739,127.7.19.125:40721,127.7.19.124:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 12:30:23.122112 29819 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:30:23.122308 29819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:23.122325 29819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:23.122345 29819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:23.123427 29819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:23.123471 29819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:23.123485 29819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:23.123498 29819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:23.123510 29819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:23.124919 29819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46739,127.7.19.125:40721,127.7.19.124:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.126.keytab
--principal=kudu/127.7.19.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.126:46739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.19.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:23.125093 29819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:23.125257 29819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:23.130326 29824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:23.130336 29827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:23.130400 29819 server_base.cc:1034] running on GCE node
W20260928 12:30:23.130362 29825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:23.130793 29819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:23.131036 29819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:23.132179 29819 hybrid_clock.cc:648] HybridClock initialized: now 1790598623132153 us; error 37 us; skew 500 ppm
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:23.134716 29819 init.cc:377] Logged in from keytab as kudu/127.7.19.126@KRBTEST.COM (short username kudu)
I20260928 12:30:23.135155 29819 webserver.cc:460] Webserver started at http://127.7.19.126:36493/ using document root <none> and password file <none>
I20260928 12:30:23.135392 29819 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:23.135437 29819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:23.135593 29819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:23.136605 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data/instance:
uuid: "1b244ee1a24242b894170eb1d0018ecd"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "00d94fdda1db67d5eb846a0634203291"
server_key_iv: "4563250f90ec2cc3ba1850205c51ed30"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.137032 29819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal/instance:
uuid: "1b244ee1a24242b894170eb1d0018ecd"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "00d94fdda1db67d5eb846a0634203291"
server_key_iv: "4563250f90ec2cc3ba1850205c51ed30"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.138403 29819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:23.139103 29834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:23.139288 29819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:23.139374 29819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal
uuid: "1b244ee1a24242b894170eb1d0018ecd"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "00d94fdda1db67d5eb846a0634203291"
server_key_iv: "4563250f90ec2cc3ba1850205c51ed30"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.139441 29819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:23.159498 29819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:23.161842 29819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:23.161979 29819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:23.166121 29819 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.126:46739
I20260928 12:30:23.166179 29886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.126:46739 every 8 connection(s)
I20260928 12:30:23.166524 29819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/data/info.pb
I20260928 12:30:23.166980 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29819
I20260928 12:30:23.167065 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-0/wal/instance
I20260928 12:30:23.167264 7245 external_mini_cluster.cc:1428] Setting key 2af365f78bf14dffc1ae402c1e0a18bb
I20260928 12:30:23.167621 29887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:23.169214 29887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.19.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.125@KRBTEST.COM" created.
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:23.181800 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.169928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.7.19.126:46739)
0928 12:30:23.170085 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.170096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.170170 (+ 74us) server_negotiation.cc:407] Connection header received
0928 12:30:23.170720 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.170722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.170744 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.170837 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.171366 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.172157 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.173099 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.173398 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.173524 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.178311 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.178324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.178324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.178330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.180191 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.180587 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.180590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.180591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.180719 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.180927 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.180928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.180929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.181137 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.181225 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.181425 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.181565 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 12:30:23.181803 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.169890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46739 (local address 127.0.0.1:42294)
0928 12:30:23.170051 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.170092 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.170562 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.170855 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.170858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.171123 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.171267 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.171271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.172266 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.172267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.173003 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.173006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.173509 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.173510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.173571 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.174111 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.174119 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.178197 (+ 4078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.180287 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.180294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.180296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.180491 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.180774 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.180774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.180775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.180858 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.181217 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.181222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.181283 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.181394 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.181628 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":99,"thread_start_us":73,"threads_started":1}
I20260928 12:30:23.182830 29887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:23.183209 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.183068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50634)
0928 12:30:23.183122 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.183175 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 12:30:23.183363 29835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
I20260928 12:30:23.183847 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.183740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50644)
0928 12:30:23.183791 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.183823 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 12:30:23.184042 29887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:40721: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab.
Entry for principal kudu/127.7.19.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab.
I20260928 12:30:23.195399 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:40721
--webserver_interface=127.7.19.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab
--principal=kudu/127.7.19.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46739,127.7.19.125:40721,127.7.19.124:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:30:23.233292 29887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } attempt: 1
I20260928 12:30:23.233832 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.233681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50650)
0928 12:30:23.233751 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.233797 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 12:30:23.234207 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.234119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50666)
0928 12:30:23.234161 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.234189 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:30:23.234344 29887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:40721: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
W20260928 12:30:23.276970 29894 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:30:23.277179 29894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:23.277197 29894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:23.277218 29894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:23.278425 29894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:23.278466 29894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:23.278481 29894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:23.278522 29894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:23.278553 29894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:23.280009 29894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46739,127.7.19.125:40721,127.7.19.124:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.125.keytab
--principal=kudu/127.7.19.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.125:40721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.19.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:23.280258 29894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:23.280539 29894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:23.285868 29899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:23.285885 29900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:23.286126 29902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:23.286360 29894 server_base.cc:1034] running on GCE node
I20260928 12:30:23.286692 29894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:23.287086 29894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:23.288236 29894 hybrid_clock.cc:648] HybridClock initialized: now 1790598623288213 us; error 40 us; skew 500 ppm
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:23.291237 29894 init.cc:377] Logged in from keytab as kudu/127.7.19.125@KRBTEST.COM (short username kudu)
I20260928 12:30:23.291672 29894 webserver.cc:460] Webserver started at http://127.7.19.125:46551/ using document root <none> and password file <none>
I20260928 12:30:23.291920 29894 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:23.291990 29894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:23.292145 29894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:23.293262 29894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data/instance:
uuid: "ca06a4d6d2d44581baeb3d80c9c84d75"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "b627dc11f77980faf3f86aacf5c4cb5f"
server_key_iv: "0ec82dacf8c1715615411eee0dcc64e3"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.293633 29894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal/instance:
uuid: "ca06a4d6d2d44581baeb3d80c9c84d75"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "b627dc11f77980faf3f86aacf5c4cb5f"
server_key_iv: "0ec82dacf8c1715615411eee0dcc64e3"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.294970 29894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:23.295663 29909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:23.296217 29894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 12:30:23.296360 29894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal
uuid: "ca06a4d6d2d44581baeb3d80c9c84d75"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "b627dc11f77980faf3f86aacf5c4cb5f"
server_key_iv: "0ec82dacf8c1715615411eee0dcc64e3"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.296439 29894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:23.302510 29887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } attempt: 2
I20260928 12:30:23.302913 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.302785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50676)
0928 12:30:23.302844 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.302884 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:30:23.303256 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.303174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50678)
0928 12:30:23.303216 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.303240 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 12:30:23.303345 29887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.125:40721: Network error: Client connection negotiation failed: client connection to 127.7.19.125:40721: connect: Connection refused (error 111)
I20260928 12:30:23.313560 29894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:23.315851 29894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:23.315964 29894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:23.319788 29894 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.125:40721
I20260928 12:30:23.319839 29961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.125:40721 every 8 connection(s)
I20260928 12:30:23.320171 29894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/data/info.pb
I20260928 12:30:23.321214 29962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:23.322748 29962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:23.330166 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29894
I20260928 12:30:23.330265 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-1/wal/instance
I20260928 12:30:23.330473 7245 external_mini_cluster.cc:1428] Setting key 9c0df63bdd53aad0d9d24086dfeee175
I20260928 12:30:23.331233 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.323389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46739 (local address 127.0.0.1:42310)
0928 12:30:23.323544 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.323586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.324046 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.324246 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.324249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.324488 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.324715 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.324721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.325350 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.325350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.326131 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.326133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.326587 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.326588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.326684 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.327203 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.327211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.328488 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.329881 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.329889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.329891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.330112 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.330340 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.330341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.330341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.330448 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.330745 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.330750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.330886 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.330973 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.331083 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 12:30:23.331264 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.323443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.7.19.126:46739)
0928 12:30:23.323498 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.323500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.323667 (+ 167us) server_negotiation.cc:407] Connection header received
0928 12:30:23.324125 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.324126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.324136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.324195 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.324798 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.325261 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.326203 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.326499 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.326557 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.328575 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.328577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.328577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.328583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.329802 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.330183 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.330185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.330186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.330254 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.330516 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.330517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.330517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.330674 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.330750 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.331023 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.331121 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 12:30:23.331825 29962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.19.124@KRBTEST.COM; defaulting to no policy
I20260928 12:30:23.339516 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.332109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50694)
0928 12:30:23.332167 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.332180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.332230 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.332449 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.332450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.332606 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.332687 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.332688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.333499 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.333501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.334396 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.334399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.334853 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.334854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.334937 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.335291 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.335296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.336565 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.338486 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.338487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.338487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.338647 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.338868 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.338868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.338869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.338922 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.339201 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.339202 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.339238 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.339317 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.339410 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 12:30:23.339604 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.332144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.7.19.125:40721)
0928 12:30:23.332288 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.332293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.332317 (+ 24us) server_negotiation.cc:407] Connection header received
0928 12:30:23.332354 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.332356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.332366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.332428 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.332754 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.333411 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.334487 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.334779 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.334837 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.336664 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.336691 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.336691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.336698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.338397 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.338740 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.338743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.338743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.338798 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.338992 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.338993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.338993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.339131 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.339221 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.339339 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.339476 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260928 12:30:23.340257 29962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:23.340662 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.340506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55898)
0928 12:30:23.340578 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.340627 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.19.124@KRBTEST.COM" created.
W20260928 12:30:23.340843 29911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
I20260928 12:30:23.342222 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.342124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55912)
0928 12:30:23.342172 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.342197 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 12:30:23.342355 29962 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44717: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.19.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab.
Entry for principal kudu/127.7.19.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab.
I20260928 12:30:23.356725 7245 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMIVMOs/build/release/bin/kudu
/tmp/dist-test-taskMIVMOs/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:44717
--webserver_interface=127.7.19.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab
--principal=kudu/127.7.19.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.19.84:46369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.19.126:46739,127.7.19.125:40721,127.7.19.124:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 12:30:23.391609 29962 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } attempt: 1
I20260928 12:30:23.392186 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.392030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55922)
0928 12:30:23.392099 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.392148 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 12:30:23.392558 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.392470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55932)
0928 12:30:23.392515 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.392541 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 12:30:23.392694 29962 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44717: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
I20260928 12:30:23.394475 29887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } attempt: 3
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:23.401523 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.394747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50708)
0928 12:30:23.394795 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.394805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.394880 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.395010 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.395011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.395175 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.395242 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.395244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.395956 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.395957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.396489 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.396490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.396938 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.396939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.397022 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.397503 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.397508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.398687 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.400551 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.400552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.400553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.400776 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.400981 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.400982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.400982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.401038 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.401294 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.401295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.401335 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.401386 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.401458 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 12:30:23.401573 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.394817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.7.19.125:40721)
0928 12:30:23.394878 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.394880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.394891 (+ 11us) server_negotiation.cc:407] Connection header received
0928 12:30:23.394940 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.394941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.394952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.394996 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.395371 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.395871 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.396575 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.396863 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.396940 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.398834 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.398836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.398836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.398844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.400467 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.400852 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.400853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.400854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.400916 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.401099 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.401100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.401100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.401233 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.401309 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.401409 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.401485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 12:30:23.402026 29887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
I20260928 12:30:23.402380 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.402259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55942)
0928 12:30:23.402324 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.402351 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 12:30:23.402709 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.402627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55950)
0928 12:30:23.402666 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.402689 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 12:30:23.402802 29887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44717: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
I20260928 12:30:23.433938 29887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } attempt: 1
I20260928 12:30:23.434368 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.434240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55952)
0928 12:30:23.434297 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.434340 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:30:23.434693 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.434598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55964)
0928 12:30:23.434642 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.434674 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 12:30:23.434784 29887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44717: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
W20260928 12:30:23.438340 29969 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 12:30:23.438532 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 12:30:23.438550 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 12:30:23.438616 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 12:30:23.439733 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 12:30:23.439772 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 12:30:23.439815 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 12:30:23.439847 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 12:30:23.439869 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 12:30:23.441334 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.19.84:46369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.19.126:46739,127.7.19.125:40721,127.7.19.124:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/krb5kdc/kudu_127.7.19.124.keytab
--principal=kudu/127.7.19.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.19.124:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.19.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 10:59:12 UTC on ccycloud.quasar-zdyxwl.root.comops.site
build id 4676954
I20260928 12:30:23.441586 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 12:30:23.441813 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 12:30:23.446802 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:23.446859 29969 server_base.cc:1034] running on GCE node
W20260928 12:30:23.446832 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 12:30:23.446803 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 12:30:23.447196 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 12:30:23.447451 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 12:30:23.448592 29969 hybrid_clock.cc:648] HybridClock initialized: now 1790598623448572 us; error 36 us; skew 500 ppm
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 12:30:23.450974 29969 init.cc:377] Logged in from keytab as kudu/127.7.19.124@KRBTEST.COM (short username kudu)
I20260928 12:30:23.451406 29969 webserver.cc:460] Webserver started at http://127.7.19.124:37195/ using document root <none> and password file <none>
I20260928 12:30:23.451638 29969 fs_manager.cc:359] Metadata directory not provided
I20260928 12:30:23.451706 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 12:30:23.451864 29969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 12:30:23.452972 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data/instance:
uuid: "8bea17cef4b94114b13eb5be98302fc8"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "ec5183b470fe7a0c4e5062c7c4b6aeb4"
server_key_iv: "4090a4ed6d8a87ad6dd0ebe9d05da77c"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.453336 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal/instance:
uuid: "8bea17cef4b94114b13eb5be98302fc8"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "ec5183b470fe7a0c4e5062c7c4b6aeb4"
server_key_iv: "4090a4ed6d8a87ad6dd0ebe9d05da77c"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.454608 29969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 12:30:23.455276 29984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 12:30:23.455446 29969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 12:30:23.455541 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data,/tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal
uuid: "8bea17cef4b94114b13eb5be98302fc8"
format_stamp: "Formatted at 2026-09-28 12:30:23 on dist-test-slave-cb7x"
server_key: "ec5183b470fe7a0c4e5062c7c4b6aeb4"
server_key_iv: "4090a4ed6d8a87ad6dd0ebe9d05da77c"
server_key_version: "encryptionkey@0"
I20260928 12:30:23.455621 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 12:30:23.460865 29962 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } attempt: 2
I20260928 12:30:23.461261 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.461114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55980)
0928 12:30:23.461166 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.461236 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 12:30:23.461616 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.461522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:55990)
0928 12:30:23.461567 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.461598 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 12:30:23.461701 29962 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.19.124:44717: Network error: Client connection negotiation failed: client connection to 127.7.19.124:44717: connect: Connection refused (error 111)
I20260928 12:30:23.466084 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 12:30:23.467826 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 12:30:23.467980 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 12:30:23.472062 29969 rpc_server.cc:307] RPC server started. Bound to: 127.7.19.124:44717
I20260928 12:30:23.472095 30036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.19.124:44717 every 8 connection(s)
I20260928 12:30:23.472474 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/data/info.pb
I20260928 12:30:23.473457 30037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 12:30:23.474965 30037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
I20260928 12:30:23.480998 7245 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMIVMOs/build/release/bin/kudu as pid 29969
I20260928 12:30:23.481112 7245 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMIVMOs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790598357699237-7245-0/minicluster-data/master-2/wal/instance
I20260928 12:30:23.481340 7245 external_mini_cluster.cc:1428] Setting key c67ba99e5ad45026647a48edee9c849e
I20260928 12:30:23.483871 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.475629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.7.19.126:46739)
0928 12:30:23.475709 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.475711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.475868 (+ 157us) server_negotiation.cc:407] Connection header received
0928 12:30:23.476290 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.476291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.476301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.476342 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.476972 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.477441 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.478372 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.478549 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.478600 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.480610 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.480611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.480611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.480619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.482481 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.482939 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.482940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.482940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.483004 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.483241 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.483242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.483243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.483377 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.483445 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.483698 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.483774 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 12:30:23.484112 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.475585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.126:46739 (local address 127.0.0.1:42326)
0928 12:30:23.475743 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.475777 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.476211 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.476396 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.476399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.476638 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.476898 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.476902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.477519 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.477520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.478289 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.478290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.478634 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.478635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.478735 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.479189 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.479197 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.480529 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.482559 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.482567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.482569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.482869 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.483073 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.483073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.483073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.483178 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.483443 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.483446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.483565 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.483648 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.483759 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260928 12:30:23.484654 30037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:23.490504 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.481749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.7.19.124:44717)
0928 12:30:23.481900 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.481905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.481946 (+ 41us) server_negotiation.cc:407] Connection header received
0928 12:30:23.481990 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.481992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.482007 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.482117 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.482838 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.483844 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.484865 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.485231 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.485316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.487560 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.487568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.487568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.487574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.489264 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.489625 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.489628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.489629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.489693 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.489894 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.489895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.489895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.490042 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.490189 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.490276 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.490353 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":98,"threads_started":1}
I20260928 12:30:23.491559 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.484947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.125:40721 (local address 127.0.0.1:50724)
0928 12:30:23.485008 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.485017 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.485050 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.485171 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.485172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.485307 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.485353 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.485354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.485966 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.485967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.486451 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.486453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.486857 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.486858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.486932 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.487351 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.487356 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.488769 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.490425 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.490426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.490426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.490660 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.490915 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.490916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.490916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.490966 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.491324 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.491325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.491369 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.491439 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.491503 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 12:30:23.491673 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.485019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.7.19.125:40721)
0928 12:30:23.485069 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.485071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.485091 (+ 20us) server_negotiation.cc:407] Connection header received
0928 12:30:23.485115 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.485116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.485125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.485170 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.485443 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.485891 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.486539 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.486765 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.486856 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.488886 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.488888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.488888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.488896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.490320 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.490777 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.490779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.490779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.490844 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.491058 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.491058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.491059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.491196 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.491270 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.491479 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.491564 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 12:30:23.492231 30037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.126@KRBTEST.COM
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.124@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:30:23.498889 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.491332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.7.19.126:46739)
0928 12:30:23.491375 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.491377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.491386 (+ 9us) server_negotiation.cc:407] Connection header received
0928 12:30:23.491418 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.491419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.491428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.491475 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.492082 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.492774 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.493442 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.493653 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.493729 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.496054 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.496056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.496056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.496063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.497749 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.498102 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.498104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.498104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.498160 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.498368 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.498369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.498369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.498485 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.498662 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.498710 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.498781 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:30:23.499017 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.492495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:56010)
0928 12:30:23.492753 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.492763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.492827 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.492951 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.492951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.493078 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.493101 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.493102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.493704 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.493705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.494156 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.494157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.494525 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.494526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.494574 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.494933 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.494937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.496385 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.498003 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.498004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.498004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.498164 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.498370 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.498371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.498371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.498424 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.498657 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.498658 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.498702 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.498788 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.498879 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260928 12:30:23.499244 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.492545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.7.19.124:44717)
0928 12:30:23.492584 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.492586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.492830 (+ 244us) server_negotiation.cc:407] Connection header received
0928 12:30:23.492884 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.492885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.492894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.492947 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.493178 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.493616 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.494232 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.494443 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.494525 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.496475 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.496476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.496476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.496483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.497931 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.498249 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.498250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.498251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.498298 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.498481 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.498481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.498482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.498606 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.498664 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.498816 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.498925 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":18}
I20260928 12:30:23.500891 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: Bootstrap starting.
I20260928 12:30:23.501863 30037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:23.502202 30037 log.cc:824] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:23.503043 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: No bootstrap required, opened a new log
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.19.125@KRBTEST.COM
I20260928 12:30:23.504758 30037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.504918 30037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bea17cef4b94114b13eb5be98302fc8, State: Initialized, Role: FOLLOWER
I20260928 12:30:23.505007 30037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.505327 30037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:23.505294 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.505421 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.507290 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.499639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.7.19.125:40721)
0928 12:30:23.499677 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.499679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.499687 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:23.499732 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.499732 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.499742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.499788 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.500326 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.500846 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.501737 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.501929 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.502010 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.504369 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.504370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.504370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.504377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.506154 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.506477 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.506478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.506479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.506562 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.506747 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.506748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.506749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.506913 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.507076 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.507113 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.507182 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:30:23.507555 30054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:23.507615 30054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:23.508469 7245 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 12:30:23.509919 29887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } attempt: 2
I20260928 12:30:23.509478 29861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1b244ee1a24242b894170eb1d0018ecd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42330
I20260928 12:30:23.507534 29969 master_runner.cc:186] Error getting master registration for 127.7.19.126:46739: OK, instance_id { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" instance_seqno: 1790598623164934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:30:23.510370 29969 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.126@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:30:23.517704 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.510337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.7.19.124:44717)
0928 12:30:23.510375 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.510377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.510384 (+ 7us) server_negotiation.cc:407] Connection header received
0928 12:30:23.510404 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.510405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.510413 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.510465 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.510761 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.511463 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.512430 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.512752 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.512815 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.514824 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.514825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.514826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.514832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.516448 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.516856 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.516857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.516857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.516914 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.517120 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.517121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.517121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.517272 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.517343 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.517533 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.517616 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 12:30:23.517724 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.510128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:56024)
0928 12:30:23.510211 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.510222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.510257 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.510480 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.510481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.510638 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.510679 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.510681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.511549 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.511550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.512346 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.512348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.512828 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.512829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.512891 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.513291 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.513297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.514731 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.516534 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.516535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.516535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.516777 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.516986 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.516987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.516987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.517053 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.517348 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.517349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.517408 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.517503 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.517630 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 12:30:23.519229 29887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd: Bootstrap starting.
I20260928 12:30:23.519941 29887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:23.520248 29887 log.cc:824] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:23.521086 29887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd: No bootstrap required, opened a new log
I20260928 12:30:23.522400 29887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.522541 29887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b244ee1a24242b894170eb1d0018ecd, State: Initialized, Role: FOLLOWER
I20260928 12:30:23.522624 29887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.522941 29887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:23.522956 29861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 12:30:23.522917 30057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.523034 29861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:23.523099 30057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.523978 29861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.524464 30011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b244ee1a24242b894170eb1d0018ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bea17cef4b94114b13eb5be98302fc8"
I20260928 12:30:23.524593 30011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [term 0 FOLLOWER]: Advancing to term 1
W20260928 12:30:23.525024 30068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:23.525130 30068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:23.525616 30011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b244ee1a24242b894170eb1d0018ecd in term 1.
I20260928 12:30:23.525856 29836 leader_election.cc:302] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b244ee1a24242b894170eb1d0018ecd, 8bea17cef4b94114b13eb5be98302fc8; no voters:
I20260928 12:30:23.525864 29861 leader_election.cc:288] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [CANDIDATE]: Term 1 election: Requested vote from peers ca06a4d6d2d44581baeb3d80c9c84d75 (127.7.19.125:40721), 8bea17cef4b94114b13eb5be98302fc8 (127.7.19.124:44717)
I20260928 12:30:23.525521 29819 master_runner.cc:186] Error getting master registration for 127.7.19.125:40721: OK, instance_id { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" instance_seqno: 1790598623318712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 12:30:23.527848 29819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 12:30:23.527956 30057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 12:30:23.528086 30057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1b244ee1a24242b894170eb1d0018ecd, State: Running, Role: LEADER
I20260928 12:30:23.528164 30057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.528548 30057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b244ee1a24242b894170eb1d0018ecd. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.528632 30057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:23.528952 30077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 12:30:23.529167 30077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 12:30:23.530638 30011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [term 1 FOLLOWER]: Refusing update from remote peer 1b244ee1a24242b894170eb1d0018ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:23.531221 30075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 12:30:23.532619 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b244ee1a24242b894170eb1d0018ecd. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.532737 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.538295 30057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.538380 30057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:23.538471 30078 mvcc.cc:205] Tried to move back new op lower bound from 7334291961979179008 to 7334291961971904512. Current Snapshot: MvccSnapshot[applied={T|T < 7334291961979179008}]
I20260928 12:30:23.538643 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.538720 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.539047 30079 mvcc.cc:205] Tried to move back new op lower bound from 7334291961979179008 to 7334291961971904512. Current Snapshot: MvccSnapshot[applied={T|T < 7334291961979179008}]
I20260928 12:30:23.549156 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.537159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.7.19.125:40721)
0928 12:30:23.537205 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.537207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.538480 (+ 1273us) server_negotiation.cc:407] Connection header received
0928 12:30:23.538589 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.538590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.538599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.538628 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.539228 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.539946 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.541853 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.542178 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.542217 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.544923 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.544925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.544925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.544932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.547513 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.548088 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.548089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.548090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.548160 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.548438 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.548439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.548440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.548575 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.548643 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.548987 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.549063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:30:23.549470 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.536905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.7.19.124:44717)
0928 12:30:23.536949 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.536951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.537269 (+ 318us) server_negotiation.cc:407] Connection header received
0928 12:30:23.537388 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.537389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.537400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.537449 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.538501 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.539201 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.540235 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.540529 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.540566 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.545493 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.545494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.545495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.545501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.548049 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.548553 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.548554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.548554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.548614 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.548851 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.548852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.548852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.548982 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.549067 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.549277 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.549372 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 12:30:23.550206 30077 catalog_manager.cc:1347] Generated new cluster ID: 264ec12b0ebf41ab952be81e3533c73b
I20260928 12:30:23.550274 30077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 12:30:23.550289 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.527067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.7.19.126:46739)
0928 12:30:23.527108 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.527111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.537282 (+ 10171us) server_negotiation.cc:407] Connection header received
0928 12:30:23.537439 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.537440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.537450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.537480 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.540000 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.540658 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.542191 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.542474 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.542517 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.545466 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.545467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.545467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.545474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.547287 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.548708 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.548710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.548710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.548761 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.549072 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.549073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.549074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.549214 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.549297 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.550059 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.550138 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 12:30:23.552824 29962 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } attempt: 3
Sep 28 12:30:23 dist-test-slave-cb7x krb5kdc[29806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790598623, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.19.125@KRBTEST.COM for kudu/127.7.19.124@KRBTEST.COM
I20260928 12:30:23.561195 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.553035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.19.124:44717 (local address 127.0.0.1:56048)
0928 12:30:23.553109 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 12:30:23.553119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 12:30:23.553152 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 12:30:23.553387 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 12:30:23.553387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 12:30:23.553535 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 12:30:23.553575 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.553577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.554461 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.554462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.555273 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 12:30:23.555276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.555815 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.555816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 12:30:23.555885 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.556332 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 12:30:23.556348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 12:30:23.557906 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 12:30:23.559985 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.559986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.559987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.560193 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.560472 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 12:30:23.560472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 12:30:23.560473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 12:30:23.560527 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 12:30:23.560913 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 12:30:23.560914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 12:30:23.560965 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 12:30:23.561011 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 12:30:23.561098 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 12:30:23.561705 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:23.553215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.7.19.124:44717)
0928 12:30:23.553262 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.553264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.553270 (+ 6us) server_negotiation.cc:407] Connection header received
0928 12:30:23.553287 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.553288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.553307 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.553351 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.553660 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.554373 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.555391 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:23.555691 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:23.555748 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:23.558018 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 12:30:23.558019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 12:30:23.558020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 12:30:23.558026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 12:30:23.559887 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.560279 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.560280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.560280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.560357 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 12:30:23.560599 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 12:30:23.560600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 12:30:23.560601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 12:30:23.560831 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 12:30:23.561396 (+ 565us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:23.561468 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 12:30:23.561578 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 12:30:23.563239 29962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: Bootstrap starting.
I20260928 12:30:23.563966 29962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: Neither blocks nor log segments found. Creating new log.
I20260928 12:30:23.564272 29962 log.cc:824] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: Log is configured to *not* fsync() on all Append() calls
I20260928 12:30:23.574433 29962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: No bootstrap required, opened a new log
I20260928 12:30:23.576821 29962 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.577214 29962 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca06a4d6d2d44581baeb3d80c9c84d75, State: Initialized, Role: FOLLOWER
I20260928 12:30:23.577399 29962 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } }
I20260928 12:30:23.577847 30077 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 12:30:23.577805 29962 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 12:30:23.577875 29936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b244ee1a24242b894170eb1d0018ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75"
I20260928 12:30:23.577910 29935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 12:30:23.579203 29935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [term 1 FOLLOWER]: Refusing update from remote peer 1b244ee1a24242b894170eb1d0018ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 12:30:23.577765 30086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.579252 30086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.579483 30075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 12:30:23.580480 30097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 12:30:23.580544 30097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 12:30:23.580658 29936 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b244ee1a24242b894170eb1d0018ecd in term 1.
I20260928 12:30:23.579497 30077 catalog_manager.cc:1504] Loading token signing keys...
I20260928 12:30:23.582065 30086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b244ee1a24242b894170eb1d0018ecd. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.582173 30086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.582091 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.582274 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 12:30:23.587364 30075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b244ee1a24242b894170eb1d0018ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b244ee1a24242b894170eb1d0018ecd" member_type: VOTER last_known_addr { host: "127.7.19.126" port: 46739 } } peers { permanent_uuid: "ca06a4d6d2d44581baeb3d80c9c84d75" member_type: VOTER last_known_addr { host: "127.7.19.125" port: 40721 } } peers { permanent_uuid: "8bea17cef4b94114b13eb5be98302fc8" member_type: VOTER last_known_addr { host: "127.7.19.124" port: 44717 } } }
I20260928 12:30:23.587463 30075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd [sys.catalog]: This master's current role is: LEADER
I20260928 12:30:23.589021 30077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b244ee1a24242b894170eb1d0018ecd: Generated new TSK 0
I20260928 12:30:23.589197 30077 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 12:30:23.614065 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.612415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.7.19.125:40721)
0928 12:30:23.612461 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.612464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.613408 (+ 944us) server_negotiation.cc:407] Connection header received
0928 12:30:23.613513 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.613514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.613525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.613555 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.613983 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50748: BlockingRecv error: recv got EOF from 127.0.0.1:50748 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 12:30:23.614933 30041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.612848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.7.19.124:44717)
0928 12:30:23.612886 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.612888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.614451 (+ 1563us) server_negotiation.cc:407] Connection header received
0928 12:30:23.614543 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.614544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.614554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.614583 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.614877 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56054: BlockingRecv error: recv got EOF from 127.0.0.1:56054 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 12:30:23.615759 29891 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.613041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.7.19.126:46739)
0928 12:30:23.613076 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.613078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.615236 (+ 2158us) server_negotiation.cc:407] Connection header received
0928 12:30:23.615356 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.615357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.615367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.615396 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.615692 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42350: BlockingRecv error: recv got EOF from 127.0.0.1:42350 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:30:23.618157 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.617327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.7.19.125:40721)
0928 12:30:23.617366 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.617368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.617683 (+ 315us) server_negotiation.cc:407] Connection header received
0928 12:30:23.617784 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.617785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.617802 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.617830 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.618113 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50760: BlockingRecv error: recv got EOF from 127.0.0.1:50760 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:30:23.621093 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.620350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.7.19.125:40721)
0928 12:30:23.620380 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.620382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.620580 (+ 198us) server_negotiation.cc:407] Connection header received
0928 12:30:23.620708 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.620709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.620719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.620747 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.621052 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50776: BlockingRecv error: recv got EOF from 127.0.0.1:50776 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 12:30:23.629671 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.628924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.7.19.125:40721)
0928 12:30:23.628960 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.628961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.629175 (+ 214us) server_negotiation.cc:407] Connection header received
0928 12:30:23.629273 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.629274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.629284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.629312 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.629622 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50792: BlockingRecv error: recv got EOF from 127.0.0.1:50792 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 12:30:23.635646 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.634915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.7.19.125:40721)
0928 12:30:23.634948 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.634950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.635143 (+ 193us) server_negotiation.cc:407] Connection header received
0928 12:30:23.635248 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.635249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.635259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.635288 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.635586 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50806: BlockingRecv error: recv got EOF from 127.0.0.1:50806 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:30:23.645723 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.644911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.7.19.125:40721)
0928 12:30:23.644958 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.644959 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.645185 (+ 226us) server_negotiation.cc:407] Connection header received
0928 12:30:23.645286 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.645287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.645298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.645328 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.645673 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50818: BlockingRecv error: recv got EOF from 127.0.0.1:50818 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:30:23.663883 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.663038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.7.19.125:40721)
0928 12:30:23.663075 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.663077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.663329 (+ 252us) server_negotiation.cc:407] Connection header received
0928 12:30:23.663430 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.663431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.663442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.663470 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.663833 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50824: BlockingRecv error: recv got EOF from 127.0.0.1:50824 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 12:30:23.698315 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.697412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.7.19.125:40721)
0928 12:30:23.697466 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.697469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.697734 (+ 265us) server_negotiation.cc:407] Connection header received
0928 12:30:23.697835 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.697836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.697848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.697876 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.698254 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50828: BlockingRecv error: recv got EOF from 127.0.0.1:50828 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 12:30:23.764627 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.763864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.7.19.125:40721)
0928 12:30:23.763954 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.763956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.764074 (+ 118us) server_negotiation.cc:407] Connection header received
0928 12:30:23.764135 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.764136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.764147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.764189 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.764555 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50830: BlockingRecv error: recv got EOF from 127.0.0.1:50830 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 12:30:23.894501 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:23.893794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.7.19.125:40721)
0928 12:30:23.893864 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:23.893866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:23.893985 (+ 119us) server_negotiation.cc:407] Connection header received
0928 12:30:23.894036 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:23.894037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:23.894047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:23.894099 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:23.894442 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50840: BlockingRecv error: recv got EOF from 127.0.0.1:50840 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 12:30:24.152412 29966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 12:30:24.151692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.7.19.125:40721)
0928 12:30:24.151773 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:24.151775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:24.151844 (+ 69us) server_negotiation.cc:407] Connection header received
0928 12:30:24.151888 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:24.151889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:24.151900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:24.151952 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 12:30:24.152329 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50852: BlockingRecv error: recv got EOF from 127.0.0.1:50852 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 12:30:24.508031 30054 catalog_manager.cc:1259] Loaded cluster ID: 264ec12b0ebf41ab952be81e3533c73b
I20260928 12:30:24.508131 30054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: loading cluster ID for follower catalog manager: success
I20260928 12:30:24.508999 30054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: acquiring CA information for follower catalog manager: success
I20260928 12:30:24.509323 30054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bea17cef4b94114b13eb5be98302fc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:30:24.580991 30097 catalog_manager.cc:1259] Loaded cluster ID: 264ec12b0ebf41ab952be81e3533c73b
I20260928 12:30:24.581070 30097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: loading cluster ID for follower catalog manager: success
I20260928 12:30:24.581895 30097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: acquiring CA information for follower catalog manager: success
I20260928 12:30:24.582222 30097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca06a4d6d2d44581baeb3d80c9c84d75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 12:30:24.668004 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:24.665572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.7.19.125:40721)
0928 12:30:24.665699 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:24.665701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:24.665751 (+ 50us) server_negotiation.cc:407] Connection header received
0928 12:30:24.665820 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:24.665820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:24.665869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:24.665937 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:24.666073 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.666570 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.667316 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.667507 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.667561 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:24.667692 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:24.667800 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:24.667862 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:24.667920 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 12:30:24.667938 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260928 12:30:24.672135 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:24.669746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.7.19.124:44717)
0928 12:30:24.669859 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:24.669862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:24.669870 (+ 8us) server_negotiation.cc:407] Connection header received
0928 12:30:24.669924 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:24.669925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:24.669961 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:24.670054 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:24.670153 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.670623 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.671370 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.671566 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.671639 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:24.671812 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:24.671903 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:24.671980 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:24.672021 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 12:30:24.672043 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260928 12:30:24.676328 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:24.673752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.7.19.125:40721)
0928 12:30:24.673808 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:24.673810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:24.673888 (+ 78us) server_negotiation.cc:407] Connection header received
0928 12:30:24.673965 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:24.673967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:24.673979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:24.674036 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:24.674164 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.674633 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.675633 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.675928 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.675996 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:24.676115 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:24.676145 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:24.676189 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:24.676235 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 12:30:24.676253 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 12:30:24.677037 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:24.673812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.7.19.126:46739)
0928 12:30:24.673960 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:24.673962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:24.674041 (+ 79us) server_negotiation.cc:407] Connection header received
0928 12:30:24.674113 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:24.674114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:24.674162 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:24.674265 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:24.674362 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.674988 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.675888 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.676198 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.676255 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:24.676711 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:24.676813 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:24.676876 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:24.676917 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 12:30:24.676946 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
I20260928 12:30:24.677453 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 12:30:24.673766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.7.19.124:44717)
0928 12:30:24.673911 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 12:30:24.673912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 12:30:24.674704 (+ 792us) server_negotiation.cc:407] Connection header received
0928 12:30:24.674724 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 12:30:24.674725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 12:30:24.674733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 12:30:24.674791 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 12:30:24.674943 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.675386 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.676611 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 12:30:24.676904 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 12:30:24.676965 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 12:30:24.677124 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 12:30:24.677157 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 12:30:24.677216 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 12:30:24.677360 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 12:30:24.677387 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 12:30:24.688236 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29819
I20260928 12:30:24.693482 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29894
I20260928 12:30:24.698243 7245 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMIVMOs/build/release/bin/kudu with pid 29969
2026-09-28T12:30:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2820 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5620 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267008 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST